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Executive Summary 


With support from the U.S. Department of Labor’s Office of Disability Employment 
Policy (ODEP), the National Collaborative on Workforce and Disability for Youth 
(NCWD/Youth) conducted the Youth Service Professionals’ Knowledge, Skills, and 
Abilities Professional Development Demonstration and Evaluation Project (YSP/KSA 
Demo) to determine the value of a professional development system for youth service 
professionals in order to improve program quality and service delivery for all youth, 
including youth with disabilities. 


Following a competitive nationwide application process, NCWD/Youth selected three 
sites to participate in the YSP/KSA Demo, including: Baltimore City Mayor’s Office of 
Employment Development, Youth Services Division — Baltimore, MD; City of Los 
Angeles Economic and Workforce Development Department — Los Angeles, CA; and 
New York State Department of Labor — Albany, NY. Each site consisted of at least three 
youth serving agencies that worked together through the demonstration project to build 
staff capacity and improve youth outcomes within their geographic region. As a part of 
the YSP/KSA Demo, each cross-system collaboration: 

¢ worked with cross-system partners to assess professional development needs 
and develop shared goals and strategies for supporting professional 
development. 

* engaged their staff in completing four full-day cross-system trainings aligned to 
the professional development needs identified in the assessment. 

* assigned two staff to serve as cadre members who assisted NCWD/Youth 
trainers with delivering the training and provided follow-up support to the trainees 
between sessions and during the follow-up period. 

* partnered with a post-secondary institution to provide credits or a credential to 
staff who completed the training. 

* participated in post-training technical assistance designed to support the trainees 
in applying what they learned and to assist the organizations in building a strong 
culture of professional development. 


The YSP/KSA Demo is part of an ongoing professional development initiative by 
NCWD/Youth. The initiative unfolded in a series of stages, beginning with the 
identification and validation of ten competency areas. Next, NCWD/Youth developed 
assessment tools for youth service professionals and organizations aligned to the 
competencies. Finally, NCWD/Youth focused on designing targeted professional 
development opportunities in order to build individual and organizational capacity, 
resulting in the creation of the YSP/KSA Training Modules for delivering in-person 
training. The YSP/KSA Training Modules provide a training curriculum aligned to the 
competencies that youth service professionals need to connect all youth, including 
those with disabilities, to opportunities for work, further education, and independent 
living (More information about the initiative, competencies, and modules is available at 


http://www.ncwd-youth.info/issues/professional-development/). 


Staff at each site completed the YSP/KSA self-assessment at the start of demonstration 
project. Using the assessment data, NCWD/Youth supported administrators at each site 
to select four of the eight training modules based on the areas of greatest need as well 
as organizational priorities across the three agencies at each site. Each site 
participated in four one-day training sessions—one day per module. Additionally, 
NCWD/Youth worked with site administrators to identify intended outcomes and 
corresponding behavioral practices to improve services for youth. To evaluate the 
quality, relevance, and effectiveness of the professional development, participants were 
asked to complete a pre-survey, a post-survey at the conclusion of each training 
session, and surveys six months and 12 months after their participation. Additionally, 
site administrators were interviewed at six months and 12 months post training. A 
summary of key findings is outlined below. 


NCWD/Youth contracted an independent third party evaluator to conduct an external 
evaluation of the YSP/KSA Demo. The evaluation was designed to examine the effects 
of professional development on youth service professionals’ practices, organizational 
culture, and outcomes for youth. Based on Kirkpatrick’s Four Level Evaluation Model, 
the evaluators collected and analyzed data for each of four levels — 1) Reaction; 2) 
Learning; 3) Behavior; and 4) Results. Participants’ reaction and learning (Kirkpatrick 
Levels one and two) were evaluated during, and at the conclusion of, the training 
delivery (a short recap of these results is included in this executive summary, complete 
results can be found in the interim evaluation report, “Pre-Post Survey Results from Site 
Trainings”). At six-month and 12-month intervals post training delivery, data was 
collected from training participants and site administrators regarding the behavior and 
results (Kirkpatrick Levels three and four). In broad terms, Levels one and two examine 
participants’ acquisition of skills, while Levels three and four examine participants’ 
application of those skills. 


Key Findings 


Pre/Post Training Delivery Evaluation Phase (summary of interim evaluation 
report) 


Reaction (Kirkpatrick Level One): Participants were asked to rate—using a five-point 
Likert scale—the usefulness and quality of each of the four trainings they attended ina 
post-training survey. Scores on the usefulness and quality of the modules ranged from 
78-100% selecting “Agree” or “Strongly Agree”. These responses were largely positive 
with the vast majority or participants in each session selecting “Agree” or “Strongly 
Agree’ for the surveyed items including: 

e “l understand how | can use what | have learned from this training module in my 
work” 
“The content from this training module is relevant to my work” 
“| would recommend this training module to others” 
“| found the facilitation style to be engaging” 
“| found the training module to be well-organized” 


Learning (Kirkpatrick Level Two): To evaluate the extent of learning, participants 
were asked to report their confidence in various skills covered in each training module 
before and after the training was delivered using a 5-point Likert scale ranging from “Not 
Confident” to “Very Confident”. To understand the relevance to their daily work, 
participants were also asked to report whether they conducted related activities “Never,” 
“Occasionally,” or “Often.” Overall, the results from all of the trainings in Baltimore, Los 
Angeles, and New York show that participants increased their confidence in their 
abilities and their knowledge with respect to many of the competencies addressed by 
the specific training modules selected by their sites. While the evaluation report on 
pre/post-training results provides the full picture of learning outcomes, the following is a 
sample of results from each site: 

e More Baltimore training participants reported post-training that they had a high 
degree of confidence in their ability to support employers providing work 
Opportunities to youth (95% were confident or very confident post-training 
compared to 83% pre-training), their ability to use informal strategies to assess 
youths’ strengths and interests (92% post-training compared to 83 percent pre- 
training), and their ability to assist youth in making decisions about disclosing a 
disability (86% post-training compared to 58% pre-training). 


e More Los Angeles training participants reported post-training that they had a high 
degree of confidence in their ability to create work-based learning opportunities in 
the community (75% were confident or very confident post-training compared to 
44% pre-training); their ability to implement career exploration activities (95% 
post-training compared to 61% pre-training); and their ability to assist youth in 
making decisions about disclosing a disability (90% post-training compared to 
50% pre-training). 


e More New York training participants reported post-training that they had a high 
degree of confidence in their ability to build youth’s employment seeking skills 
(88% were confident or very confident post-training compared to 74% pre- 
training); their ability to implement career exploration activities (84% post-training 
compared to 74% pre-training); and their ability to assist youth in making 
decisions about disclosing a disability (87% post-training compared to 48% pre- 
training). 


Six-Month Follow-up Evaluation Phase 


Behavior (Kirkpatrick Level Three): Individual Practices & Organizational Support 
for Implementation 


The results from the six-month follow-up evaluation show a strong level of behavior 
change indicating that participants consistently applied what they learned during the 
training. Participants reported regularly using strategies (70%), using resources (50%) 
from the training in their individual practice as well as sharing knowledge gained 
(64%) from the training with other staff in their agency, behaviors which are strong 


predictors of improved organizational capacity. Corresponding data from the 
administrator interviews highlight that the individual practices of staff were more highly 
influenced by participation in the YSP/KSA Demo Project than were the cross-agency 
connections. 


Baltimore 


At six months post-training, the majority of respondents said they are: using assessment 
information to develop youth individual service plans (71.4%), tailoring the individual 
service plans to each youth’s individual needs and strengths (78.6%), regularly 
identifying and engaging new employer partners (64.3%), regularly maintaining contact 
with existing employer partners (57.1%), and using universal design to serve youth and 
young people with disabilities. These results directly illustrate behavioral changes of 
youth service professionals who participated in the training. 


Los Angeles 


While not as strong as the Baltimore results, Los Angeles participants report regularly 
using assessment information to develop youth individual service plans (55.6%) and 
tailoring the individual service plans to each youth’s needs and strengths (55.6%). 
There is also a high frequency of identifying and documenting youths’ career interests 
(61.1%) and in engaging them in career exploration activities relevant to those interests 
(61.1%). Many respondents (55.6%) regularly help youth obtain work experiences 
relevant to their career interests, yet several respondents (38.9%) indicated that this 
practice is not applicable to their job. More than half of the respondents (61.1%) also 
provide information to youth on career pathway options regularly. 


New York 


The practices related to identifying, engaging, and maintaining contact with employer 
partners was applicable to fewer of the respondents. Of the 80% who indicated it was 
applicable to their job, 58.3% reported that they more regularly identify and engage new 
employer partners, and 40.0% maintain some frequency of regular contact with existing 
employer partners. More respondents reported that identifying and engaging new and 
existing partners and stakeholders was applicable to their job. At least half of the 
respondents do these activities frequently. More than half (58.3%) of the respondents 
indicated that they incorporate youth voice into day-to-day youth activities or support 
others to do so all the time or at least frequently. 


While it may be tempting to compare the results from the trainings across sites, it is 
important to note the unique circumstances between the sites that make such 
comparisons less meaningful. For example, each site involved different types and levels 
of agencies in this effort. Baltimore’s cross-system partners were all within one 
department of the Mayor’s office. Los Angeles brings together departments across the 
city (Workforce Development, Education, and Children and Family Services) and New 


York brings together state-level agencies (Department of Labor, Education Department, 
and Children and Family Services) to impact one region of the state. 


Analysis of the organizational support site-based responses indicated strong 
agreement that participants felt encouraged to apply their learning and improve their 
practices. This is consistent across sites: 87.2% in Baltimore, 80.3% in Los Angeles, 
and 86.7% in New York. Within each site, responses to the items soliciting information 
about more general aspects of encouragement had higher agreement ratings than 
those that examined improving specific practices. For example, in Baltimore the items 
about encouragement to generally improve practices had higher agreement (92.9%) 
than those specifically regarding improving services to youth with disabilities (78.6%). 
In Los Angeles, the agreement ranged from 100% for the encouragement to improve 
the quality of youth service plans to 41.2% agreement that the organization has 
developed a shared resource map for working across partner agencies. The New York 
responses follow the same overall pattern where encouragement to implement 
strategies to increase stakeholder involvement received higher agreement (100%) than 
the item regarding implementing these strategies specifically to diverse populations 
(73.3%). 


The results regarding influences on the agencies’ culture of professional 
development also reflect a high degree of understanding of its critical role in behavior 
change. All three sites reported having a stronger sense of what quality professional 
development involves and how this needs to be closely related to staff job 
responsibilities rather than a general requirement for attendance at a certain number of 
training opportunities. 


12-Month Follow-Up Evaluation Phase 


Results (Kirkpatrick Level Four): Organizational Practices & Culture of 
Professional Development 

The 12-month evaluation activities focused on organizational results at each of the three 
sites, as well as the continuing behavior changes to achieve those results. The online 
training surveys and interviews with site administrators asked specifically about the 
degree to which their organizational outcomes were achieved. While the main focus of 
the 12-month survey was organizational results, questions from the six-month survey 
were repeated to determine whether behavior changes reflected in the six-month survey 
results persisted. 


An analysis of the survey results from each of the sites does reveal a high degree of 
behavior change in individual practice related to the desired results. Respondents 
were regularly engaging in individual practices identified to achieve outcomes. 
Specifically, Baltimore had an average of 66% agreement that participants had changed 
individual practices; Los Angeles had 61% agreement, and in New York 55% of survey 
respondents reported that they implemented the individual practices frequently or all of 
the time. Many trainee respondents commented on the change in organizational culture 
since the trainings, noting the improved communication between staff and program 


participants, empowerment of individual staff to do their job with confidence, higher 
levels of sensitivity to specific issues such as mental health and gender identification, 
and increased coaching and support from supervisors. 


The same pattern was evident in the organizational results. An average of 69% of 
Baltimore survey respondents agreed that their organization's work was addressing 
their site specific outcomes, 73% of Los Angeles participants agreed regarding their 
organizations, and in New York, 68% agreed that this was the case. While the sites 
varied in the extent to which they achieved their outcomes, all made progress toward 
them. 


The site administrator interviews serve to confirm information about the changes in 
behavior related to intended outcomes and achievement of results at their sites. 
Consistent with the six-month interviews, at 12 months post-training site administrators 
reported that staff are implementing what they learned from the YSP/KSA Training. 
However, sites continue to report that participants struggle to make connections across 
the partner agencies. While more may be done to strengthen connections across the 
partner agencies, respondents to the online survey did report that there is nowa 
common language and increased cooperation among the partners. 


Administrators also shared key lessons regarding training participant selection and 
engagement. All three sites noted that they would have liked more time to consider 
which staff should attend and to do more thoughtful selection of the specific training 
modules in order to make their participation in the YSP/KSA Demo more effective. 


Conclusion 


The overall purpose of the YSP/KSA Demo is to examine the “value of and further grow 
a professional development system for youth service professionals to improve program 
quality and service delivery for all youth, including youth with disabilities”. 


Evaluation results reflect that the training materials and resources were high quality and 
will Support participants in continuing to implement strategies they learned. Staff from 
the three sites participating in the YSP/KSA Demo increased both their knowledge and 
their confidence in the competencies from the training experience and, as a result, are 
changing their individual practices to serve youth. Data also demonstrates that there is 
a strong correlation between organizational support and a related behavior change on 
the part of staff. Organizationally, administrators reported an appreciation for the quality 
professional development provided and each has taken some steps within their 
agencies to continue the changed behaviors and outcomes from the YSP/KSA Demo. 


The results described in this report validate that quality professional development that is 
meaningful and relevant leads to sustained changes in behavior and practices of youth 
service professionals. Those changes are enhanced when supported by an 
organization’s administration and a professional development culture. When quality 


training is coupled with organizational support, the resulting changes in daily practices 
lead to improved services and increased opportunities for youth. 


Overview of the Project & Evaluation 


The Systems Improvement Group at the Institute on Community Integration (ICI), 
University of Minnesota was contracted by the National Collaborative on Workforce and 
Disability for Youth (NCWD/Youth), housed at the Institute for Educational Leadership, 
to conduct an external evaluation of the Youth Service Professionals’ Knowledge, Skills, 
and Abilities Professional Development Demonstration and Evaluation Project 
(YSP/KSA Demo). 


About the Demonstation Project 


With support from the U.S. Department of Labor’s Office of Disability Employment 
Policy (ODEP), NCWD/Youth conducted a professional development project to 
demonstrate the value of and further grow a professional development system for youth 
service professionals to improve program quality and service delivery for all youth, 
including youth with disabilities. 


The objectives of the YSP/KSA Demo were: 

¢ To demonstrate the components of high quality professional development in the 
workforce development system to provide lessons that state and local systems 
can replicate. 

¢ To deliver high quality professional development training and technical 
assistance to youth service professionals (YSPs) across a variety of systems. 

¢ To evaluate the effect of professional development on YSPs’ competencies and 
practices, as well as the organization’s policies and professional development 
culture. 

* To increase alignment of professional development competencies of YSPs 
across youth systems. 

¢ To inform federal, state, and local policy development to promote an effective 
foundation and infrastructure for the professional development of YSPs. 


Following a competitive nationwide application process, NCWD/Youth selected three 
sites to participate in the YSP/KSA Demo. The three sites are Baltimore City Mayor’s 
Office of Employment Development (MOED), Youth Services Division — Baltimore, MD; 
City of Los Angeles Economic and Workforce Development Department — Los Angeles, 
CA; and New York State Department of Labor — Albany, NY. Each cross-system site 
consisted of at least three youth serving agencies that were working together through 
the demonstration project to build staff capacity and improve youth outcomes within 
their geographic region. As part of the demonstration, each site had to: 
¢ work with cross-system partners to assess professional development needs and 
develop shared goals and strategies for supporting professional development. 
* engage their staff in completing four full-day cross-agency trainings aligned to the 
professional development needs identified in the assessment. 
* assign two staff to serve as cadre members in training who assist NCWD/Youth 
trainers with delivering the training and provide follow-up support to the trainees 
in-between training sessions and during the follow-up period. 


* partner with a post-secondary institution to provide credits or a credential to staff 
who completed the training. 

* participate in post-training technical assistance designed to support the trainees 
in applying what they learned through the training sessions and to assist the 
organizations in building a strong culture of professional development. 


The YSP/KSA Demo was part of an ongoing professional development initiative by 
NCWD/Youth that included identifying key competency areas, assessing youth service 
professional and organizational needs in those areas, and building individual and 
organizational capacity. Staff at each site completed the YSP/KSA self-assessment at 
the start of demonstration project. Then, NCWD/Youth worked with site administrators 
to select four of the eight YSP/KSA Training Modules based on the areas of greatest 
need and interest indicated in these assessments as well as organizational priorities 
across the three agencies at each site. The YSP/KSA Training Modules were designed 
to build key competencies that youth service professionals need to connect all youth, 
including those with disabilities, to opportunities for work, further education, and 
independent living. (More information about the initiative, competencies, and modules is 


available at http://www.ncwd-youth. info/issues/professional-development/). 


About the Evaluation 


The evaluation of the YSP/KSA Demo was designed to examine the effects of 
professional development on youth service professionals’ practices, organizational 
culture, and outcomes for youth. Based on Kirkpatrick’s Four Level Evaluation Model, 
the evaluators collect and analyze data for each of the four Kirkpatrick levels — 1) 
Reaction; 2) Learning; 3) Behavior; and 4) Results. The Reaction and Learning levels 
were evaluated during, and at the conclusion of the training phase of the demonstration 
project. In pre and post-surveys, participants reported the module contents and delivery 
were high quality. They also indicated an increase in confidence in their skills and 
knowledge of the topics covered in the four modules at their site. Complete findings 
from the pre-post training surveys are described in the interim evaluation report “Pre- 
Post Survey Results from Site Trainings.” 


This final report focuses on findings from the evaluation of the third and fourth 
Kirkpatrick levels, Behavior and Results, which were conducted during the follow-up 
phase of the demonstration project. The follow-up phase evaluation consisted of six- 
month and 12-month post-training online surveys and semi-structured telephone 
interviews with the organization administrators. The primary data source used for the 
analysis reflected in this report is the online surveys. Qualitative data from the site 
administrator interviews were included when there were correlating themes and or 
contextual information that supported the results. 


About the Follow-Up Phase 


The follow-up phase evaluation was intended to determine what effect participating in 
the YSP/KSA Demo has had on each demonstration site with respect to: 


1) Implementing and strengthening the practices of staff who serve youth, with a 
particular focus on practices specifically targeted for improvement by each site 
(as defined in their project logic model); and 

2) Achieving site-specific targeted outcomes (as defined in a project logic model). 


At the start of the follow-up phase, IEL guided each site through a process of updating 
their demonstration project logic model. The purpose of the logic model was to help 
sites define their long-term and mid-term outcomes for using the knowledge, skills, and 
relationships developed through the YSP/KSA Demo and to identify action steps 
(referred to as drivers in the Kirkpatrick model) that the organizations would take to 
support their staff in implementing new practices to achieve the outcomes. This process 
was Critical to ensuring that each sites’ organization administrators recognized their role 
in encouraging and supporting their staff in applying what they learned through the 
YSP/KSA training. 
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Data Collection Methods & Analyses 


Online Survey of Trainees 


Prior to the six-month follow-up activities, attendance records for the training module 
sessions for all three YSP/KSA Demo sites were reviewed so that only those who 
attended each training module would be surveyed. Prior to both the six- and 12-month 
follow-up activities, the contact information for all those who attended the trainings was 
updated so that surveys would reach as many participants as possible and any changes 
in position or organization would be captured. A separate survey was created for each 
site based on the logic model they developed and the specific practices in that logic 
model that the administrators wanted to target for organizational improvement. Once 
these surveys were finalized, a request to complete the survey was distributed to the 
Staff lists at each demonstration site. For the six-month survey, a total of 94 trainees 
were provided the link to the online survey that included 32 for Baltimore, 37 for Los 
Angeles, and 25 from New York. For the 12-month survey, a total of 93 trainees were 
provided the link to the online survey that included 31 for Baltimore, 37 for Los Angeles, 
and 25 from New York. The survey remained open for several weeks and reminders 
were sent to prompt responses, but, in order to protect the validity of the evaluation 
responses, participation was completely voluntary and not required as part of the 
YSP/KSA Demo. 


Each of the site-specific surveys included a mix of multiple choice questions as well as 
some open-ended items. These items were designed to solicit feedback on the extent 
to which participants in the training sessions were regularly implementing the 
knowledge and strategies learned through the trainings in their individual practice as 
well as their perceptions whether their organizations had achieved the outcomes the 
desired improvement outlined in their logic model. The survey results were analyzed for 
each of the items and included quantitative and qualitative approaches depending on 
the particular items. 


Semi- Structured Interviews with Site Administrators 


At six months, telephone interviews were conducted with site administrators to solicit 
information about 1) organizational practices, 2) perceptions about the extent to which 
strategies from the training modules were being implemented at each of the sites, and 
3) changes to the culture of professional development in the agencies participating. A 
total of seven (7) interviews were conducted and included two (2) for Baltimore, two (2) 
for Los Angeles, and three (3) for New York. At 12 months, telephone interviews were 
conducted with site administrators to solicit information about the degree to which the 
long-term outcomes of their logic models were achieved. To facilitate scheduling and to 
foster a richer discussion, the telephone interviews were conducted with three 
administrators for each site and with all of the administrators as a group when possible. 
In both Baltimore and LA, three administrators were interviewed in a single telephone 
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call. In New York, two administrators were interviewed together and a third was 
interviewed at a later time due to a personal emergency. A set of interview questions 
was developed for each site based on the logic models created to guide the 
implementation of the training strategies at the participating agencies. Results from 
these interviews were analyzed and themes were developed for each site based on 
individual responses to the interview questions. Where these themes correlated to the 
results from the online surveys, information was included in this evaluation report. 
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Baltimore Follow-Up Evaluation Results 


Baltimore Site Overview 


The Baltimore site consists of participants from multiple agencies within the Baltimore 
City Mayor’s Office of Employment Development, Youth Services Division. These 
include The Career Academy/Baltimore City Schools, and the Housing Authority’s 
Successfully Moving Youth to Work agencies. As part of their participation in the 
YSP/KSA Demo, staff from these agencies participated in the Knowledge of the Field 
training module, and three additional training modules selected by the agency 
administrators based on their organizational needs: 1) Communication with Youth, 2) 
Assessment and Individualized Planning, and 3) Employer Relations. After staff had 
completed all four training modules, the agency administrators identified relevant long- 
term outcomes (Results) and related short-term goals (Behavior Changes) they 
expected to achieve during the post-training follow-up phase. The purpose of setting 
these outcome goals following the training was to focus efforts of the administrators and 
additional technical assistance provided by NCWD/Youth on supporting staff in applying 
what they learned from the training to improve some specific practices organization- 
wide. 


Baltimore’s four long-term outcomes and related short-term goals were: 
1. Improve quality and individualization of the youth service plans 
e Use assessment information to develop individual service plans 
e More frequently review and update the plans 
2. Increase and sustain employer engagement 
e identify and engage new employer partners 
e take steps to maintain and sustain relationships with existing employer 
partners 
3. Improve services to and engagement of youth with disabilities 
e Identify and incorporate youth’s disability related needs into individual 
service plans 
e Connect youth with disabilities to disability services in the community 
when needed 
e Use universal design strategies and accommodations to better engage 
youth with disabilities 
4. Improve individual staff practices based on what each trainee defined as their 
action steps for implementing what they learned during the training 
e apply various practices and knowledge gained in the training to their work 
as defined by each individual trainee through action steps 


Survey items were developed to collect data on the extent to which participants were 


implementing or strengthening these specific practices and the extent to which the 
desired long-term results were achieved. 
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Six-Month Follow-Up Data Collection on Behavior Change 


A total of 32 participants were sent a request to complete the six-month follow up survey 
regarding application of what they learned through the YSP/KSA trainings (behavior 
change). Of those, 15 provided responses to the survey, yielding a response rate of 
47%. Three (3) of the respondents did not complete the survey and provided answers 
to only some of the items. The responses were provided from staff across each of the 
partner agencies. Only two (2) respondents reported that they had changed positions 
since the trainings and 11 of the 15 respondents indicated they worked directly with 
youth and young adults in their current positions. 


Six-Month Findings 
Individual Practices—Trainees’ Descriptions of Behavior Changes 


A set of open-ended survey items provided the opportunity for respondents to indicate 
any individual actions they had taken in relation to the specific practices that were 
targeted for improvement in the demonstration site logic model. 
Nearly all reported that they had done something in the following 
areas: 1) improve the quality and individualization of youth 
service plans, 2) increase and sustain employer engagement, 
and 3) improve services to and engagement of youth with 
disabilities. 


Responses to the actions taken to improve the quality and individualization of youth 
service plans ranged from general statements about being more focused on young 
people’s needs to descriptions of adjustments in the process for assessing those needs. 
For example, respondents described how their organization had made changes to the 
way youth were interviewed so that it would yield more in-depth and accurate 
information upfront, thereby improving the individualization of the 
service plan. Others described efforts to strengthen partnerships 
with employers so that youth and young people had more 
options for placements that suited their individual needs and 
interests. 


Those responding to how they have worked to increase and 
sustain employer engagement indicated these efforts were about 
stasis doing more as well as doing things differently. In general they 
have done more partnering, sent out more correspondence, and invited more feedback. 
As far as doing things differently, some examples include: inviting employer partners to 
be involved in the mock interview process, adjusting the way they explain their program 

to employers, and inviting them to come to see the agency's program at work. 


The comments provided regarding improving service to and engagement of youth with 
disabilities echo those provided in response to improving the quality of service for all 
youth. Namely, being aware of individual needs, being more patient, and asking 
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questions differently. As far as services, respondents reported that they were 
identifying more partners, resources, and services specifically to support young people 
with disabilities. A few described how they had focused on conducting assessments 
aimed at the needs of youth with disabilities and had made accommodations to support 
their career preparation. 


Individual Practices—Trainees’ Ratings of Behavior Changes 


Training participants were asked to indicate the extent to which they had taken some 
general actions (e.g., used strategies, shared knowledge with other staff) as well as 
implemented targeted practices since the trainings. These 12 items were rated using a 
Likert Scale of four options regarding frequency (“All of the Time”, “Frequently”, 
“Infrequently”, and “Rarely”), as well as an option to indicate the particular practice 
“Does Not Apply to My Job.” For the purposes of analysis, the ratings of “All of the 
Time” and “Frequently” were combined into an overall percentage that indicates more 
regular/consistent implementation of the specific practice, which was termed “Doing 
Regularly.” Of note, very few (one or two only) selected the option of “Rarely” for any of 
the items. The results for each of these survey items is depicted in Figure 1 which 
reflects the combined responses of “All of the Time” and “Frequently” as “Doing 
Regularly.” 


Almost all training participants responding to the survey have regularly used strategies 
and/or resources from the trainings (100% using strategies; 92.9% using resources). 
The majority, though fewer also indicated they have regularly shared the knowledge 
they’ve gained with others (78.6%) and/or communicated with staff from the partner 
agencies (64.3%). Most said they are using assessment information to develop youth 
individual service plans (71.4%) and tailoring the individual service plans to each 
youth’s individual needs and strengths (78.6%). 
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Figure 1. Individual Practices Item Results by Percentage: Baltimore 
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Fewer participants, though still the majority, indicated they are regularly implementing 
the practices related to employer partnerships. About sixty-four percent of the 
respondents (64.3%) indicated that they regularly identify and engage new employer 
partners, and only 57.1% regularly maintain regular contact with existing employer 
partners. 


Fewer respondents also indicated that they regularly carried out practices related 
specifically to youth and young people with disabilities. More than half (57.1%) regularly 
incorporated the disability related needs of youth into her/his individual service plan, and 
50.0% regularly connected youth to disability services available in the community. 
Similarly, respondents indicated that they use universal design to serve youth and 
young people with disabilities (57.1%). Correspondingly, 14.3% of respondents 
indicated they infrequently incorporated disability related needs of youth into her/his 
individual service plan and another 7.1% do this rarely. Equal percentages of 
respondents infrequently or rarely connected youth to disability services available in the 
community (14.3%). Regarding the use of universal design, 14.3% indicated they rarely 
do this. For all three of these items, there was a higher percentage of respondents 
indicating these practices were not applicable to their job as compared to other items 
(21.4%, 21.4%, and 28.6% respectively). 
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Organizational Support for Behavior Change 


Respondents were asked to provide their level of agreement to a set of survey items 
regarding whether their organization encourages employees to apply the knowledge 
and skills gained as a result of the training. These items included a four-point 
agreement scale (“Strongly Agree”, “Agree”, “Disagree”, “Strongly Disagree”) as well as 
the option to indicate that the item “Does Not Apply to my Job”. For the purposes of 
analysis, the “Strongly Agree” and “Agree” responses were combined to provide an 
overall “Agreement” percentage and the “Disagree” and “Strongly Disagree” responses 
were combined to provide an overall “Disagreement” percentage. These results are 
displayed in Figure 2. 


There was a high level of agreement - over 90% - for three of these five items. The 
highest level of agreement was with the items related to whether the organization 
encourages employees to make changes or improvements to practices, to implement 
new strategies and, more specifically, to increase and sustain employer engagement 
(92.9% for each item). Slightly fewer respondents agreed that their organization 
encourages me to improve the quality of the youths’ individual service plans and 
provides encouragement to improve how | engage youth will disabilities (78.6%) for 
each item). As with the individual practice items, the items related to students with 
disabilities also included more responses (14%) stated that these organizational 
practices were not applicable to their job. Similarly, 14% of respondents indicated that 
the item about improving individual service plans does not apply to their job. 


Figure 2. Organizational Practice Items Results by Percentage: Baltimore 
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Action Steps Completed 


Prior to the invitation to participate in the six-month survey, staff that attended the 
trainings were provided a list of action steps that they indicated they planned to take. 


dls 


Those action steps came from the surveys administered immediately after their initial 
YSP/KSA trainings. Of those responding to whether they completed any of their action 
steps, 93% indicated they had. The types of actions reported ranged from more 
outreach to employers, ensuring that they emphasize the needs of youth with disabilities 
and encouraging other staff to do so, as well as identifying specific services for those 
youth and young people. Others had taken action regarding marketing their program 
and improving their interactions with youth — building rapport, asking meaningful 
questions. The one (1) respondent who indicated s/he did not complete any of their 
action steps made a commitment to doing so in the near future. 


Changes Made at Organization/General Observations 


Several respondents provided comments to two (2) open-ended Survey items regarding 
their general observations as well as observations on changes made in how youth are 
served. Some were specific to processes that have been changed (e.g., redesigned 
youth intake assessment tool, risk management assessment tools), and activities 
implemented to sustain the learning (e.g., monthly refresher courses, reminders from 
supervisors about the training resources). Input on specific changes in progress 
regarding youth assessment processes were also noted during the site administrator 
interviews. The goal to streamline the youth assessment and intake process and related 
forms was specifically reported in the interviews. 


Many trainee survey respondents commented on the change in organizational culture 
since the trainings, noting the improved communication among staff and program 
participants, empowerment of individual staff to do their job with confidence, higher 
levels of sensitivity to specific issues such as mental health and gender identification, 
and increased coaching and support from supervisors. The site administrator interviews 
also pointed to a general sense of viewing the work as more than “paperwork” and a 
renewed commitment to working more meaningfully with the youth and young adults 
served by the agencies. One survey comment indicated that there could be even more 
done to address the needs of youth with disabilities, but overall it was reported that staff 
are getting better information upfront and using that to develop more individualized 
youth service plans. 


12-Month Follow-Up Data Collection 


A total of 31 participants were sent a request to complete the twelve-month follow up 
survey regarding regular implementation of the knowledge they acquired through the 
YSP/KSA trainings. Of those, 15 provided responses to the survey, yielding a response 
rate of 48%. The responses were provided from staff across each of the partner 
agencies. Only one (1) respondent reported that s/he had changed positions since the 
trainings and 12 of the 15 respondents indicated they worked directly with youth and 
young adults in their current positions. 
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12-Month Findings 


Improved Individual Practices 


Through the online survey, training participants were asked to indicate the extent to 
which they regularly implement targeted practices since the trainings. While the main 
focus on the 12-month survey was organizational results, questions about targeted 
individual practices that were asked at the six-month point were repeated at the 12- 
month point to determine whether more participants had changed their behavior by the 
12-month point and whether behavior changes that were reflected in the six-month 
survey results persisted. 


These seven (7) individual practice items were rated using a Likert Scale of agreement 
(“Strongly Disagree”, “Disagree”, “Agree”, and “Strongly Agree”), as well as an option to 
indicate the particular practice “Does Not Apply to My Job.” For the purposes of 
reporting results, the ratings are described in terms of overall agreement (i.e., combined 
responses of “Agree” and “Strongly Agree”) and overall disagreement (i.e., combined 
responses of “Disagree” and “Strongly Disagree”). Of note, respondents reported 
higher agreement than disagreement across all seven (7) of the items that they were 
regularly implementing the individual practices. The detailed results for each of these 
survey items is depicted in Figure 3. 


The individual practice that most online survey respondents strongly agreed that they 
were doing regularly is using assessment information to develop youth’s individual 
service plans (53%). This was closely followed by tailoring the youth’s individual service 
plan to individual needs and strengths to which 40% of the respondents strongly agreed 
and 27% agreed. Results from the group interview with three site administrators 
supported this perception in that they described changes to the assessment process 
that better matched youths’ needs and ultimately resulted in greater individualization of 
service plans. Most respondents agreed that they were regularly implementing the 
individual practices related specifically to supporting youth with disabilities. These 
practices are: incorporating disability related needs into individual services plans, 
connecting youth with disabilities to services available in the community, and using 
universal design and accommodations to better serve youth with disabilities. Overall 
agreement for these items was high (60%, 60%, and 53% respectively). The 
percentages of those selecting “Does Not Apply to My Job” were higher for each of 
those items as well, with 40% of respondents indicating that the use of universal design 
and accommodations does not apply to their job. 
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Figure 3. Change in Individual Practice: Baltimore 
(by percent agreement) 
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Results of the group interviews support the findings of the online survey responses 
regarding improved use of assessment information to individualize youth service plans. 
Administrators described changes underway in the application and assessment process 
that facilitate effective matching of youth to employment opportunities. They also 
echoed the need to better serve youth with disabilities, but did describe that the 
improvements in the assessment process help staff get information that identified 
additional support needs for students (e.g., tutoring). One administrator reported that 
they now had one staff as the “point person” for youth with disabilities and this may be 
why there was a higher percentage of survey respondents who indicated that using 
universal design and accommodations and incorporating disability needs into youth 
service plans did not apply to their job. 


Regarding individual practices related to employer partners, the majority of respondents 
agreed that they were regularly identifying and engaging new employer partners (80%) 
and that they maintained contact with existing employer partners (66%). While there 
was a high level of agreement with maintaining contact with existing employer partners, 
this item was the one with the highest level of respondents “Strongly Disagreeing” that 
they do this regularly. This may be an area for follow up with the site administrators and 
their staff to determine why this is the case and if any supports could be provided to 
assist staff in doing this more regularly. 


The site administrators also identified that engagement of employer partners was an 
area that needed continued improvement and agreed they were working on more 
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effective outreach about their programs — both to youth and employers. One 
administrator described increased marketing to youth about the range of employers they 
work with in an effort to engage them in the program. Another described the need to let 
employer partners know that there is flexibility in working with youth. Specifically, being 
explicit that if there is a need, they could do on-site coaching for those youth and that 
the relationship with their agency continues after the youth are placed. Keeping these 
relationships effective requires continued communication and the group of 
administrators agreed that they are working on this and will continue to do so. 


Organizational Results — Achieving Long-term Outcomes 


Respondents to the online survey were asked to provide their level of agreement to a 
set of items regarding whether their organization had achieved the long-term outcomes 
that their site had identified in the logic model. These items included a four-point 
agreement scale (“Strongly Agree”, “Agree”, “Disagree”, “Strongly Disagree”). As with 
the individual practice items, there was higher agreement than disagreement with the 
statements regarding achievement of desired outcomes for the organizations 
participating in the YSP/KSA Demo. Details of the results for all the organizational 
outcomes items are displayed in Figure 4. 


Respondents indicated the highest degree of agreement that their organizations had 
increased the number of employer partners (40% “Strongly Agree”), and highest overall 
agreement was regarding the improved quality of youth service plans (87%). There was 
the most disagreement regarding the results for youth with disabilities. Nearly half of 
the respondents (46%) indicated disagreement with the statement that more youth with 
disabilities are connected to disability services matching their needs and 40% disagreed 
that youth with disabilities were better served as a result of universal design strategies. 
Of the 40% who disagreed with this item, 20% of those strongly disagreed. In analyzing 
the trainee responses to the desired results for the Baltimore demonstration site, the 
majority of respondents agreed that these results were achieved. There is, of course, 
variance in the degree of agreement and, as a result, how youth with disabilities are 
supported may be an area of further exploration. 
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Figure 4. Organizational Results - Achieving Targeted Outcomes: Baltimore 
(by percent agreement) 
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In addition to the forced-choice rating items, the survey included an item inviting 
respondents to describe any additional changes in their organization. Three (3) 
respondents answered and provided specific information on how their organizations had 
changed. One cited improvements to communication and assessment strategies and 
re-enforced that the youth service plans were more individualized than they were prior 
to participating in the YSP/KSA Demo. Another pointed out that there was increased 
partnering with other agencies and the third commented that staff were making an effort 
to apply what they learned through the training modules in their daily work. 


In the group interview, administrators offered their overall perspectives regarding the 
results of participation in the YSP/KSA Demo. These included a sense of renewed 
energy on the part of staff and a feeling of validation for the work they do. One 
administrator noted that it gave newer staff tools they needed and more seasoned staff 
some new strategies that supplemented their “tried and true” ones. All agreed that their 
staff use the tools that were introduced in the trainings and have applied them to 
improving their processes and ultimately their services to youth. 
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Los Angeles Follow-Up Evaluation Results 


Los Angeles Site Overview 


The Los Angeles site consisted of participants from the City of Los Angeles Economic & 
Workforce Development Department, the Los Angeles Unified School District, and the 
County Department of Children and Family Services. As part of their participation in the 
YSP/KSA Demo, staff from these agencies participated in the Knowledge of the Field 
training module, and three additional training modules selected by the agency 
administrators based on their organizational needs. These modules were: 1) 
Assessment and Individualized Planning, 2) Career Exploration and Workforce 
Preparation, and 3) Community Resources. After staff had completed all four training 
modules, the agency administrators identified relevant long-term outcomes (Results) 
and related short-term goals (Behavior Changes) they expected to achieve during the 
post-training follow-up phase. The purpose of setting these outcome goals following the 
training was to focus efforts of the administrators and additional technical assistance 
provided by NCWD/Youth on supporting staff in applying what they learned from the 
training to improve some specific practices organization-wide. 


Los Angeles’ four long-term outcomes and related short-term goals were: 


1. Establish a system for ongoing cross-system professional development 
a. Staff participate in cross-system professional development activities 
b. Staff connect and communicate with partner agency staff who also 
attended the training 
2. Improved quality and individualization of the youth service plans 
a. Individual service plans reflect that staff have used intake information to 
select services and strategies appropriate to each youth’s personal goals 
b. More youth are receiving services/programming that match their individual 
goals 
3. Increased engagement of youth in hands-on work experiences based on their 
interests 
a. Youth placed in a wider variety of internships, summer jobs, trainings, 
service projects, and subsidized/unsubsidized employment opportunities 
b. Connection in youth’s individual plans between interests and their hands- 
on work experience placement 
4. Shared resource map and systemic approach to youth transition. 
a. Staff use the resource map to refer youth to services 
b. Staff use the resource map to increase partnerships 


Survey items were developed to collect data on the extent to which participants were 


implementing or strengthening these specific practices and the extent to which the 
desired long-term results were achieved. 
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Six-month Follow-Up Data Collection on Behavior Change 


A total of 37 participants were sent a request to complete the six-month follow up survey 
regarding their application of what they learned through the YSP/KSA trainings 
(behavior change). Of those, 18 provided responses to the survey items, yielding a 
response rate of 49%. The responses were provided from staff at each of the partner 
agencies. Only three (3) respondents reported that they had changed positions since 
the trainings and 14 of the 37 respondents indicated they worked directly with youth and 
young adults in their current positions. The implications of the higher number of 
respondents not working directly with youth and young adults is discussed in the 
“Individual Practices” section of the results summary. 


Six-month Findings 
Individual Practices—Trainees’ Descriptions of Behavior Changes 


A set of open-ended survey items provided the opportunity for respondents to indicate 
any individual actions they had taken in relation to the specific practices that were 
targeted for improvement in the demonstration site logic model. Nearly all respondents 
reported that they had done something in the following areas: 1) improve the quality and 
individualization of youth service plans, and 2) share resources. Respondents 
described some specific practices related to the individualization of youth service plans 
and generally how they had shared resources. In terms of the youth service plans, 
actions taken included using a variety of assessment tools and ensuring the information 
provided a comprehensive picture in order to specifically target the youth’s areas of 
interest and skills. Improvements to communication and connection with youth were 
also named. Some specific improvements to communication included: following up 
frequently and asking different questions. One (1) respondent indicated s/he was 
focusing on youth-centered planning in a general sense. With regard to the sharing of 
resources, this is being done by email as well as face-to-face and is done when a 
specific need or question arises and then a specific resource or web site is shared to 
address that need. Three (3) respondents reported that they had regularly shared 
resources prior to the trainings and this was not improved or affected as a result of the 
YSP/KSA Demo trainings in which they participated. 


Individual Practices—Trainees’ Ratings of Behavior Changes 


Training participants were asked to indicate the extent to which they had implemented 
specific individual practices since the trainings. These 16 items were rated using a 4- 
point Likert Scale regarding frequency (“All of the Time”, “Frequently”, “Infrequently”, 
and “Rarely”), as well as an option to indicate the particular practice “Does Not Apply to 
My Job”. For the purposes of analysis, the ratings of “All of the Time” and “Frequently” 
were combined into an overall percentage that indicates more regular/consistent 
implementation of the specific practice. Figure 5 displays the responses to these items 
and includes the combined responses to “All of the Time” and “Frequently” into “Doing 
More Regularly”. 
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Responses indicate that for those who found it was applicable to their job, more than 
half regularly used strategies learned (55.6%). On the related item regarding use of 
resources provided during the trainings, slightly fewer of the respondents (47.1%) 
reported doing so regularly. Most of the respondents indicated that they more regularly 
share the knowledge gained during the trainings with other staff (72.2%), and most also 
share resources with their colleagues with the same level of frequency (77.8%). 


There is some communication and connection across the staff from the partner 
agencies. Nearly half (44.4%) of the respondents reported communicating 
professionally with staff from the partnering agencies regularly, and 33.3% do so, but 
infrequently. The same percentage of respondents (44.4%) reported connecting with at 
least one of the partner agencies regularly. Half of the respondents (50%) indicated that 
they participated in cross-system professional development regularly post-trainings and 
another 33.3% reported this happens infrequently. Over half (61.1%) reported that they 
shared resources with other professionals at the partner agencies regularly. 


Figure 5. Individual Practice Items Results by Percentage: Los Angeles 


0 20 40 60 80 100 
T t t t 1 


| share resources with others within our organization. = 
I share knowledge gained during the trainings with other staff. << 

| provide information to youth on career pathway optiors. KL. —_ 

| identify and document youth's career interests.. ——L_. << 

| engage youth in career exploration activities relevant to theirinterests. —<—<<<€§ << 
| share resources with other professionals and partner agencks. —_—_—— - 


| help youth obtainwork expefiences relevant to their career interests. eee 


| tailor the individual service plans that | develop with youth to each youtl’s | 
individual needs and goals. ] 


| use assessment information to develop youth's individual service plans. ————————EEE—— 


| use strategies | learned during the trainings in my professional practice. eaoEeoaE——eEeEeEeEeEe—eeeEeee 
| participate in cross-system professional devaopment activities. ——EEEEE 

| use a resource map torefer youthto services. a 

| connect with atleast one of the partner agencies who attended the 
YSP/KSA training. | 

| 


| communicate professionaly with staf from any of the partnering agencies tS 
who attended the traning. Le ae 1 


| use a resource map toincrease partnerships. ——————— eee 


| use resources provided during the trainings in my professional practice. : SS Ss 


"= Doing Regularly "infrequently “Rarely ™ Does Not Apply to My Job 


Most respondents are regularly using assessment information to develop youth 
individual service plans (55.6%) and tailoring the individual service plans to each 
youth’s needs and strengths (55.6%). There is also a high frequency of identifying and 
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documenting youths’ career interests (61.1%) and engaging them in career exploration 
activities relevant to those interests (61.1%). Many respondents (55.6%) regularly help 
youth obtain work experiences relevant to their career interests, yet several 
respondents (38.9%) did not find this applicable to their job. More than half of the 
respondents (61.1%) also provide information to youth on career pathway options 
regularly. Responses regarding the use of a resource map to refer youth to services 
indicate that nearly half (47.1%) do this regularly and slightly fewer (38.9%) reported 
using a resource map to increase partnerships regularly. 


These responses align with information from the administrators’ interviews that provide 
context to the range in regular application of practices since the trainings. In the 
interviews, administrators noted that staff attending the trainings were varied in their 
skills and needs for professional development. This had to do with their particular role 
and the practices they were already doing or practices for which they felt the need for 
skill building. Although the trainings provided useful strategies for some, they were more 
of a refresher for others who may not have gotten as much out of them and therefore 
their responses to the online survey reflect lower percentages of regular application of 
the individual practices. When reflecting on what they may have done differently, 
administrators indicated that they would have selected a more proportionate mix of 
participants from each of the participating agencies based on the needs for skill building 
and strategies to assist them in the particular roles. In addition, data from the 
administrators’ interviews indicated that the choice of training modules might not have 
matched the needs of participating staff as closely as it could have. This may account 
for the higher percentages of respondents who indicated that the practice “Does Not 
Apply to my Job”. 


Organizational Support for Behavior Change 


Respondents were asked to provide their level of agreement to a set of survey items 
regarding whether their organization encourages employees to apply the knowledge 
and skills gained as a result of the training. These items included a four-point 
agreement scale (“Strongly Agree”, “Agree”, “Disagree”, “Strongly Disagree”) as well as 
the option to indicate that the item “Does Not Apply to my Job”. For the purposes of 
analysis, the “Strongly Agree” and “Agree" responses were combined to provide an 
overall “Agreement” percentage and the “Disagree” and “Strongly Disagree” responses 
were combined to provide an overall “Disagreement” percentage. The results for all of 
these items are displayed in Figure 6. 
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Figure 6. Organizational Practice Items by Percentage Agreement: 
Los Angeles 
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There was strong agreement related to the encouragement of quality individual 
practices by organizations. All the respondents (100%) agreed that their organization 
encouraged individuals to improve the quality of youth’s individual service plans, while 
88.9% reported that their organizations encouraged improving career exploration and 
work experience opportunities for youth and young people. Nearly ninety-five percent 
(94.4%) indicated that there was encouragement for providing information to youth on 
career pathway options. Almost all respondents (94.4%) indicated there was 
encouragement to share resources within their organization. 


There was a moderate level of agreement for the items related to encouragement to 
participate in cross-system professional development and/or to connect with partner 
agency staff who attended the training (72.2% and 70.6% respectively). Only 41.2% 
percent agreed that their organization had developed a resource map with the partner 
agencies that attended the trainings. 


When looking at the responses regarding individual practice together with the 
responses about support from the organization, there are some connections between 
and among the items. For example, the individual practice item | provide information to 
youth on career pathway options was done regularly by 84.6% of respondents and the 
organizational practice item my organization encourages employees to provide 
information to youth on career pathway options was agreed to by 94.4% of respondents. 
This indicates that this practice was encouraged and being carried out. All of the 


2? 


respondents reported that their organization encourages employees to improve the 
quality of youth's individual service plans and the individual practices being conducted 
regularly relate specifically to the youth’s interests and tailoring the service plans. This 
indicates a positive focus on youth interests and including those interests in the 
resulting service plans. While the sharing and collaboration with partners is important, 
these individual practices were more often reported to be happening infrequently or 
rarely and are not what was reported as having the strongest organizational support. 


Action Steps Completed 


Prior to the invitation to participate in the six-month survey, staff who attended the 
training were provided a list of action steps they indicated they would like to take in the 
post-survey administered immediately following their trainings. Of those responding to 
whether they completed any of their action steps (n=18), the responses were split with 
half indicating they had taken action (n=9) and half not (n=9). Those completing some of 
their action steps described sharing resources and/or identifying resources they might 
use, reviewing and re-affirming their approach to using assessment information when 
developing individual service plans, and participating in follow-up training. Of the others 
who indicated they did not complete their actions steps, two indicated they had shifted 
jobs and so the actions they originally listed no longer applied to their work. Two others 
indicated it was due to a lack of time or a lack of interest, and two didn’t provide any 
reason for not completing their action steps. 


Changes Made at Organization/General Observations 


Several respondents provided comments to two 
open-ended survey items regarding their 
observations on changes made in how youth are 
served post-trainings and general observations. 
Those respondents providing observations about 
changes to how youth are served (n=11) indicated 
that there is a level of awareness about what is 
needed and that the focus is on increasing the 
quality of the services or enhancing services. This 
was also reflected in a comment about connecting 
and collaborating with partners — the awareness is 
raised and the commitment to at least “stay in the 
loop” is there. One respondent pointed to the 
importance of individual beliefs in supporting the youth and young people and in 
particular, knowing why the work is important so that this is reflected in the services 
provided. Several of the comments provided regarding general observations indicated 
that there was a great deal of staff turnover since the trainings and this may make it 
difficult to implement what was provided. One suggestion was to have more regular 
trainings — especially for the counselors — so that strategies could be “refreshed” for 
those who originally attended and provided to those who were new since the last 
trainings were offered. Another respondent suggested that the time commitment to meet 
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and co-train was something that administrators might address and support. Without this 
message, the professional development would not be highly valued and therefore, may 
not be a priority amid the day-to-day workload. 


12-Month Follow-Up Data Collection 


A total of 37 participants were sent a request to complete the 12-month follow up survey 
regarding their application of the knowledge they acquired through the YSP/KSA 
trainings. Of those, nine (9) provided responses to the survey items, yielding a response 
rate of 24%. The responses were provided from staff at each of the partner agencies. 
Only three (3) respondents reported that they had changed positions since the trainings 
and none of these were major changes. Five of the nine respondents indicated they 
worked directly with youth and young adults in their current positions. The implications 
of the higher number of respondents not working directly with youth and young adults is 
discussed in the “Individual Practices” section of the results summary. 


12-Month Findings 


Improved Individual Practices 


Training participants were asked to indicate the extent to which they agreed that they 
were implementing specific individual practices since the trainings. While the main focus 
on the 12-month survey was organizational results, questions about targeted individual 
practices that were asked at the six-month point were repeated at the 12-month point to 
determine whether more participants had changed their behavior by the 12-month point 
and whether behavior changes that were reflected in the six-month survey results 
persisted. 


These 10 individual practice items were rated using a 4-point Likert Scale regarding 
agreement (“Strongly Disagree”, “Disagree”, “Agree”, and “Strongly Agree”), as well as 
an option to indicate the particular practice “Does Not Apply to My Job”. For the 
purposes of reporting results, the ratings are described in terms of overall agreement 
(i.e., combined responses of “Agree” and “Strongly Agree”) and overall disagreement 
(i.e., combined responses of “Disagree” and “Strongly Disagree”). Of note, none of the 
respondents “Strongly Disagreed” that they were regularly implementing the individual 
practices. The detailed results for each of these survey items is depicted in Figure 7. 
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Figure 7. Change in Individual Practice: Los Angeles 
(by percent agreement) 
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| regularly help youth obtain work experiences relevant to 
their career interests. 


| regularly engage youth in career exploration activities 
relevant to their career interests. 


| regularly provide information to youth on career pathway 
options. 


| regularly identify and document youth's career interests. 


| regularly use assessment information to develop youth's 
individual service plans. 


| regularly tailor the individual service plans that | develop 
with youth to each youth's individual needs and goals. 


| regularly connect with at least one of the partner agencies 
who attended the YSP/KSA training. 


| regularly participate in cross-system professional 
development activities. 


| regularly share resources with other professionals from the 
partner agencies. 


| regularly share resources with others in our organization. 


Strongly Disagree mDisgree mAgree mStonglyAgree mDoes Not Apply to My Job 


As indicated previously, five of the nine respondents reported that they did not work 
directly with youth and their responses to the individual practices related specifically to 
identify and document youth's career interests, engage youth in career exploration 
activities relevant to their career interests, and helping youth obtain experiences related 
to their career interests, with opportunities and incorporating their interests in individual 
service plans indicated those practices do not apply to their job (56% for each of these 
practices). Of those who felt it was a part of their job, all agreed to some extent that 
they were regularly implementing those practices. 


The three site administrators participating in the group interview provided their 
perspectives on the improved quality of youth service plans since the YSP/KSA Demo 
trainings. They noted that improved communication within and across the agencies was 
supporting the individualization of youth service plans. Managers and staff are working 
on making connections between the information gathered from youth and how this is 
used. Staff does this in practice and managers look for evidence of the practice when 
they monitor. 


Regarding cross-agency interactions, 55% indicated they regularly connect with staff at 
the partner agencies. This practice had the highest level of disagreement across the 10 
items (44%). More (77%) agreed that they regularly participate in cross-agency 
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professional development, and 77% share resources with staff at the partner agencies. 
All respondents (100%) agreed that they share resources with staff within their 
organization. This is an indication that cross-agency interactions are happening and 
demonstration project resources are being shared within and across organizations. 


The group interview data adds to the picture of cross-agency professional development. 
Administrators described that this professional development approach is continuing and 
one reported that the culture of professional development that existed in one agency 
has spread to the others. Specifically, that regular needs assessments are conducted 
and professional development is identified based on those needs, and follow-up is done 
to ensure it was useful to the participants. Another stated that even if the agency 
partners don’t necessarily go to the same professional development, they “pool 
information” about resources (e.g., upcoming trainings) and make it available to all staff 
in case it is needed. 


Connections between and among staff at the partner agencies is something that the site 
administrators saw as an area that hadn’t improved as much as others. They described 
the challenges of making those connections in the midst of full workloads and day-to- 
day activities. Calling and/or reaching out to a colleague in one of the partner agencies 
is not the “first thing” staff think of doing while trying to get their work done. One 
administrator did indicate that there were signs of improved connections in that staff 
from partner agencies sit with each other at trainings rather than isolate themselves with 
staff from their own agency. 


Organizational Results 


Respondents were asked to provide their level of agreement to a set of Survey items 
regarding whether their organization had achieved the long-term outcomes identified in 
their site’s logic model. These items included a four-point agreement scale (“Strongly 
Agree”, “Agree”, “Disagree”, “Strongly Disagree”). There was higher agreement than 
disagreement across all the statements regarding achievement of desired outcomes for 
the organizations participating in the YSP/KSA Demo. Details of the results for all the 
organizational outcomes are displayed in Figure 8. 


Analysis of the responses from the online survey correlates with the data from the 
individual practice survey items discussed previously. Specifically, the item regarding 
the desired result of building connections across partner agencies was the item with the 
highest disagreement (44%) as was the case in the individual practice item. The 
majority of respondents to the online survey agreed to some level that the set of desired 
results were achieved, and all (100%) indicated agreement that their organization 
connects youth to a wider variety of opportunities. 
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Figure 8. Organizational Results - Achieving Targeted Outcomes: Los Angeles 
(by percent agreement) 
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More youth have been participating in work 
experiences relevant to their career interests. 


The quality of youth service plans has become more 
tailored to each youth's individual needs and goals. 


Our organization offers cross-system professional 
development opportunities. 


Staff from the three partner agencies are sharing 
resources for youth more frequently. 


Our organization has built connections with at least one 
of the partner agencies who attended the YSP/KSA 
training. 


Our organization has worked with a greater variety of 
partners. 


Our organization connects youth to a wider variety of 
internships, summer jobs, training, service projects, or 
subsidized/unsubsidized employment opportunities. 


m=Strongly Disagree mDisgree mAgree mStonglyAgree sDoes Not Apply to My Job 


The data from the site administrators’ group interview indicate that they, too, feel like the 
desired results were implemented to some degree. While the specific outcome of 
developing a resource map was not achieved, there is definitely sharing of resources 
across the agencies and plans to develop a shared space where resources can be 
accessed by all of the partner agencies. This sharing is happening informally as well. 
One administrator described a benefit of the YSP/KSA Demo as helping the 
organization extend their culture of continuous improvement and the role of professional 
development in that. The training, resources, and connections have helped make their 
system more effective which will ultimately benefit the youth who are provided services. 


A 


New York Follow-Up Evaluation Results 


New York Site Overview 


The New York site consisted of participants from the New York State Department of 
Labor; the Office of Children and Family Services, Division of Juvenile Justice and 
Opportunities for Youth; and the New York State Education Department, Office of Adult 
Career and Continuing Education Services — Vocational Rehabilitation (ACCES-VR) 
and the P-12 Office of Special Education. In addition to the training module Knowledge 
of the Field, the NYS demonstration site administrators selected the following three 
modules based on their needs across state agencies: 1) Career Exploration and 
Workforce Preparation, 2) Employer Relations, and 3) Program Design, Delivery, and 
Administration. After staff had completed all four training modules, the agency 
administrators identified relevant long-term outcomes (Results) and related short-term 
goals (Behavior Changes) they expected to achieve during the post-training follow-up 
phase. The purpose of setting these outcome goals following the training was to focus 
efforts of the administrators and additional technical assistance provided by 
NCWD/Youth on supporting staff in applying what they learned from the training to 
improve some specific practices organization-wide. 


New York’s four long-term outcomes and related short-term goals were: 


1. Increased cross-system collaboration on youth workforce development 
a. Professionals from the three agencies are regularly connecting as a 
community of practice 
b. The youth related units from the three agencies demonstrate increased 
cooperation (e.g. regularly share resources and strategies) 
2. Increased cross-agency knowledge around workforce issues and best practices 
a. Professionals from the three agencies participate in activities designed to 
build cross-agency knowledge around workforce issues and best practices 
3. Increased engagement of stakeholders and partners. 
a. Staff regularly engage identify and engage new employer partners 
b. Staff regularly maintain contact with exiting employer partners 
c. Staff regularly identify and engage new partners and stakeholders 
d. Staff regularly incorporate youth voice into day-to-day youth activities 
4. Strengthen/improve various staff practices 
a. Staff apply various practices gained in the training to their work as defined 
in individual action steps 


Survey items were developed to collect data on the extent to which participants were 


implementing or strengthening these specific practices and to what extent the desired 
long-term results were achieved. 
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Six-month Follow-Up Data Collection on Behavior Change 


A total of 25 participants were sent a request to complete the six-month follow up survey 
regarding application of what they learned through the YSP/KSA trainings (behavior 
change). Of those, 16 provided responses to the survey items, yielding a response rate 
of 64%. The responses were provided from staff at each of the partner agencies. Of 
those responding, seven (7) indicated they had changed positions, organizations, or 
agencies since attending the trainings. Only three (3) respondents indicated that they 
worked directly with youth, which is not unusual based on the approach of this particular 
demonstration site. Unlike the others, New York is working across the state agency 
level in a specific region of the state. The focus on collaboration and connections at this 
level is among and between staff whose roles at their agency is not necessarily at the 
level of working directly with youth and young people, but rather supporting their 
agencies regarding those issues. 


Six-month Findings 
Individual Practices—Trainees’ Descriptions of Behavior Changes 


As was done with the other two sites, a set of open-ended survey items provided the 
opportunity for respondents to indicate any individual actions they had taken in relation 
to the specific practices that were targeted for improvement in the demonstration site 
logic model. Nearly all reported that they had done something in the following areas: 1) 
connect, collaborate, or cooperate with the staff of any of the other agencies, 2) build 
cross-agency knowledge around workforce issues and best practice, 3) increase and 
strengthen employer engagement, and 4) increase the engagement of stakeholders and 
partners. 


The majority of responses reflect that there is a level of connection and collaboration 
across the agencies. Specific activities reported include regular e-mail contact, sharing 
resources, and cross-agency presentations. One (1) respondent described the 
development of a cross-agency glossary for their website to assist families of youth with 
special needs. Many of the respondents described knowledge building activities such as 
being part of cross agency workgroups that address specific issues as well as 
developing presentations which drew from the resources across agency partners. 
There were few respondents who reported that they had done something related to 
increasing and/or strengthening employer engagement. They reported continuing to do 
this, but did not report implementing any specific strategies from the trainings or doing 
anything differently based on what they learned at the trainings. Regarding increasing 
the engagement of stakeholders and partners, most respondents indicated this was 
being done at the awareness raising level and that they were involved in efforts to 
strategize how to do this better. Two (2) reported specific activities in which they had 
Opportunities to interact and engage with stakeholders. One (1) provided technical 
assistance using the training module materials to stakeholders through a related grant; 
the other attended community roundtable events. 
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Individual Practices—Trainees’ Ratings of Behavior Changes 


Training participants were asked to indicate the extent to which they had implemented 
specific individual practices since the trainings. These 12 items were rated using a 4- 
point Likert Scale regarding frequency (“All of the Time”, “Frequently”, “Infrequently”, 
and “Rarely”), as well as an option to indicate the particular practice “Does Not Apply to 
My Job”. 

Figure 9 displays the responses to these items of reporting results, the ratings of “All of 
the Time” and “Frequently” are described together as an overall percentage that 
indicates more regular/consistent implementation (e.g. doing regularly) of the specific 
practice. 


Responses to the survey items indicate that 60.0% of the respondents use strategies 
learned during the trainings regularly while only 26.7% reported using resources from 
the trainings regularly. Respondents also reported sharing the knowledge gained during 
the training with other staff. Responses were split between those who did this regularly 
(40.0%) and those who did so infrequently (46.7%), and only one (1) person reported 
doing this rarely. 


Responses to the items about connections to partner agencies indicate that this is 
occurring, but the frequency varies depending on the level of connection. Over half of 
the respondents (53.3%) indicated that they connect with at least one of the partner 
agencies more regularly, while fewer communicate professionally or share resources 
more regularly (26.7% for each item). These activities happen, but are more infrequent. 
Specific to building cross-agency knowledge around workforce issues, responses were 
split between doing this more regularly and infrequently (46.7% and 40.0% 
respectively). 


The practices related to identifying, engaging, and maintaining contact with employer 
partners was applicable to fewer of the respondents. Of the 80% who indicated it was 
applicable to their job, 58.3% reported that they more regularly identify and engage new 
employer partners, and 40.0% maintain some frequency of regular contact with existing 
employer partners. More respondents reported that identifying and engaging new and 
existing partners and stakeholders was applicable to their job. At least half of the 
respondents do these activities frequently. 


More than half (58.3%) of the respondents indicated that they more regularly 
incorporate youth voice into day-to-day youth activities or support others to do so. 
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Figure 9. Individual Practice Items by Percentage: New York 
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 


| share resources and strategies with staff form at least one 
of the partner agencies who attended the YSP/KSA training. 
| use resources provided during the trainings in my 
professional practice. 
| communicate professionally with staff from any of the 
partnering agencies who attended the training. 
| incorporate youth voice into day-to-day youth activities or 
support others to do so. 
| maintain regular contact with existing employer partners or 
support others to do so. 
| share knowledge gained during the trainings with other 
staff. 
| connect with at least one of the partner agencies who 
attended the YSP/KSA training. 
| identify and engage new employer partners or support 
others to do so. 
| identify and engage new partners and stakeholders or 
support others to do so. 
| participate in activities designed to build cross-agency 
knowledge around workforce issues and best practices. 
| use new strategies to engage existing partners and 
stakeholders or support others to do so. 
| use strategies | learned during the trainings in my 
professional practice. 


BAIl of the Time m= Frequently a lnfrequently = Rarely = Does Not Apply to My Job 


Organizational Support for Behavior Change 


Respondents were asked to provide their level of agreement to a set of survey items 
regarding whether their organization encourages employees to apply the knowledge 
and skills gained as a result of the training. These items included a four-point 
agreement scale (“Strongly Agree”, “Agree”, “Disagree”, “Strongly Disagree”) as well as 
the option to indicate that the item “Does Not Apply to my Job”. For the purposes of 
analysis, the “Strongly Agree” and “Agree” responses were combined to provide an 
overall “Agreement” percentage and the “Disagree” and “Strongly Disagree” responses 
were combined to provide an overall “Disagreement” percentage. The results for all of 
these items are displayed in Figure 10. 
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Figure 10. Organizational Practice Items Summary: New York 
0% 20% 40% 60% 80% 100% 


My organization encourages me to implement 
strategies to increase the engagement of 
stakeholders and partners. 


My organization encourages employees to connect, 
collaborate, or cooperate with the partner agencies 
who attended the YSP/KSA training. 


My organization encourages employees to participate 
in activities designed to build cross-agency 
knowledge around workforce issues and best 
practices. 


My organization encourages me to implement 
strategies to increase and strengthen employer 
engagement with diverse youth populations. 


mAgreement mDisagreement Does Not Apply 


All of the respondents (100%) agreed that their organizations encourage the 
implementation of strategies to increase stakeholder and partner engagement. 
Respondents also agreed that their organizations encouraged connection, collaboration, 
and cooperation with the partner agencies as well as opportunities to build cross- 
agency knowledge (86.7% for each item).The efforts regarding building cross-agency 
knowledge are supported by some of the qualitative data collected through the site 
administrator interviews. Those responses indicate that there are connections 
happening and that the agencies have a common language and better awareness of 
each other’s roles and resources in supporting youth and young people. The survey 
item with the lowest percentage of agreement was related to implementing strategies to 
increase and strengthen employer engagement with diverse youth populations (73.3%). 
This item also had the highest percentage of respondents who indicated that this 
organizational practice was not applicable to their job (26.7%). 


Action Steps Completed 


Prior to the request to participate in the six-month survey, staff who attended the 
training were provided a list of action steps they previously had indicated they would like 
to take. These action steps were included on the post-survey administered immediately 
following their training sessions. Of those responding to this survey item, 73.3% 
indicated they had taken some action since the trainings. The actions described 
included keeping in touch with their peer partners from the training and partners in other 
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agencies. Some respondents included specific strategies they have implemented, such 
as, reviewing and revising some of the questions used when talking to youth and 
employer partners, developing an experience worksheet, and reviewing the agency 
program model to determine whether quality youth programs are being provided. Of 
those who responded that they had not taken any action steps, three provided reasons 
including the following: changes in the respondent's role at the agency, the respondent 
was previously doing them as part of their work, and the respondent was in the planning 
stage of implementing them. 


Changes Made at Organization/General Observations 


Responses to how the strategies presented in the trainings were used included both 
general and very specific examples. Many had generally shared the resources or used 
them to develop webinars. Two (2) respondents mentioned using the resources from 
Module 7: Employer Engagement; one named the Guideposts for 

Success; another the 10 competency areas; and one described using the overview 
information on a Workforce Development System to gain a better understanding of how 
systems relate to one another. 


Regarding shifts in how agencies are serving youth, two (2) respondents indicated that 
they were engaging youth and young people in different ways and attempting to include 
them in more aspects of the work. Many respondents indicated that they were 
connecting and sharing resources with other agencies and 
providing professional development to support service 
providers for youth and young adults. 


General observations about the trainings and how the 
strategies and resources have been used highlighted an 
appreciation for increased access to helpful resources. 
Two (2) respondents reported a desire for additional 
training and perhaps more focused training on some of the 
topics covered in the original trainings — an opportunity to 
“dive deeper” into some of the topics. Many mentioned the 
connections to the recently re-authorized federal workforce 
legislation, the Workforce Innovation and Opportunity Act 
(WIOA), indicating that the knowledge and resources they obtained through the training 
will support implementing some WIOA requirements related to serving youth and young 
people with disabilities. In their interviews, site administrators also described 
connections to both a recently funded Promoting the Readiness of Minors in 
Supplemental Security Income (PROMISE) grant and planning related to WIOA. In 
particular, site administrators had difficulty determining the extent to which the 
collaboration and communication across the partner agencies was due to the 
Opportunities provided through the YSP/KSA Demo, or to their collaborative work on 
initiatives within the PROMISE Grants, or to joint planning in preparation for 
implementing WIOA. It is likely a combination of each of these activities— the trainings, 
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the grant initiatives, and the need for implementation of WIOA — contributed to more 
frequent connections among the partner agencies. 


12-Month Follow-Up Data Collection 


A total of 25 participants were sent a request to complete the twelve-month follow up 
survey regarding implementation of the strategies and knowledge gained through the 
YSP/KSA Demo trainings. Of those, six (6) provided responses to the survey items, 
yielding a response rate of 24%. The lower response rate may have been due to a high 
proportion of job changes among the New York participants since they had completed 
the training. Of those responding, three (3) indicated they had changed positions, 
organizations, or agencies since attending the trainings. One moved from providing 
direct services to youth to not doing so, another the opposite, and the third moved out of 
her/his agency all together. In total, four (4) respondents indicated that they worked 
directly with youth, which is not unusual based on the approach of this particular 
demonstration site. Unlike the others, New York is working across the state agency 
level in a specific region of the state. The focus on collaboration and connections at this 
level is among and between staff whose roles at their agency is not necessarily at the 
level of working directly with youth and young people, but rather supporting their 
agencies regarding youth services and programs. 


Due to scheduling constraints and unexpected events, the group interview with all three 
administrators at one time was not conducted as planned. Instead one interview took 
place with two administrators and the third was conducted individually. Because of the 
low response rate from individual participants, more data from the administrator 
interviews is provided for this site in order to describe the 12-month findings as they 
relate to a culture of professional development. 


12-Month Follow-Up Findings 


Improved Individual Practices 


Training participants were asked to indicate the extent to which they regularly 
implement specific individual practices since the trainings. While the main focus on the 
12-month survey was organizational results, questions about targeted individual 
practices that were asked at the six-month point were repeated at the 12-month point to 
determine whether more participants had changed their behavior by the 12-month point 
and whether behavior changes that were reflected in the six-month survey results 
persisted. 


These eight (8) individual practice items were rated using a 4-point Likert Scale 
regarding frequency (“All of the Time”, “Frequently”, “Infrequently”, and “Rarely”), as well 
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as an option to indicate the particular practice “Does Not Apply to My Job”.+ For 
purposes of discussion, the ratings for “All of the Time” and “Frequently” are interpreted 
as the individual practices being implemented with some regularity. Details of all of the 
responses are displayed in Figure 11. 


Of the individual practices being implemented, respondents indicated that they are more 
regularly participating in activities designed to build cross-agency knowledge around 
workforce issues and best practices (67% frequently). The individual practice done 
more rarely was sharing resources with staff from partner agencies. The specific 
practices of engaging employer partners and maintaining contact with existing employer 
partners was One more respondents indicated does not apply to their job. 


Figure 11. Frequency of Individual Practice: New York (by percentage) 
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| regularly incorporate youth voice into day-to-day youth activities st 
or support others to do so. 
| regularly use new strategies to engage existing partners and =z #@=—S—=sé«S0 J 
stakeholders or support others to do so. 
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support others to do so. 


| regularly maintain contact with existing employer partners or 
support others to do so. 


| regularly identify and engage new employer partners or support 
others to do so. 


| regularly participate in activities designed to build cross-agency 
knowledge around workforce issues and best practices. 


| regularly share resources and strategies with staff from at least 
on one of the partner agencies who attended the YSP/KSA 
training. 


| regularly connect with at least one of the partner agencies who 
attended the YSP/KSA training. 


mRarely omlnfrequently mFrequently mAllofthe Time Does Not Apply to My Job 


Organizational Results — Achieving Long-term Outcomes 


Respondents were asked to provide their level of agreement to a set of survey items 
regarding whether their organization had achieved the long-term outcomes identified in 
their sites logic model. These items included a four-point agreement scale (“Strongly 
Agree’, “Agree”, “Disagree”, “Strongly Disagree”). For the purposes of discussion, the 
ratings are described in terms of overall agreement (i.e., combined responses of 
“Agree” and “Strongly Agree”) and overall disagreement (i.e., combined responses of 


1 The survey for the New York demonstration site inadvertently included the incorrect scale for the items 
regarding individual practice. Unlike the other surveys, frequency scale was included rather than an 
agreement scale. 
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“Disagree” and “Strongly Disagree”). Details of the results for all the organizational 
outcome items are displayed in Figure 12. 


All of the respondents (100%) agreed that their organization increased and deepened 
youth engagement and voice. Eighty percent (80%) agreed that their employer partners 
are more engaged with diverse populations, and 60% agreed that their organization had 
engaged new partners and stakeholders, while only 40% agreed that their 
organization’s existing partners and stakeholders were deeply engaged. Regarding 
connections across the partner agencies, there was higher agreement that staff had 
developed a shared/common language and that there was increased cooperation 
between the organizations (80% respectively), than there was that professionals from 
the partners agencies regular connect as a community of practice (40%). 


Figure 12. Organizational Results - Achieving Targeted Outcomes: New York 
(by percentage) 
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Our organization has increased and deepened youth 
engagement and voice. 


Our organization's existing partners and stakeholders are 
more deeply engaged. 


Our organization has engaged new partners and 
stakeholders. 


Our employer partners are more engaged with diverse 
populations. 


The number of employer partners has increased. 


Staff from the partner agencies have developed a 
shared/common knowledge about workforce issues and best 
practice. 

There is increased cooperation between our organization and 
at least one of the partner agencies who attended the 
YSP/KSA training. 

Professionals from the partner agencies who attended the 
YSP/KSA training are regularly connecting as a community of 
practice. 


w Strongly Disagree mw Disagree m= Agree = Strongly Agree 


As with the other demonstration sites, the results from New York respondents indicate 
that there has been some achievement of desired outcomes and, in particular, stronger 
connections across the partner agencies who are serving youth in this area of the state. 
Data from the interviews with site administrators supports the individual responses 
regarding stronger connections across the partner agencies from different systems. 
The respondents described that these connections were more at the administrator level 
and they weren't certain that the same was true for connections at the staff level. While 
the cross-system partnerships would have been established with the work required to 
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implement WIOA activities, two of the three administrators reported that their 
participation in the YSP/KSA Demo facilitated that work. 


The interviews with the three site administrators provided additional information about 
the culture of professional development and contextual factors that influenced their 
participation in the demonstration project. As reflected in the six-month data collection 
interviews, the administrators had difficulty in teasing out the specific benefits of the 
YSP/KSA Demo from the recent work on WIOA implementation. All three noted the 
high quality of the training materials and one administrator described how some of the 
materials from the YSP/KSA Demo were being incorporated into an upcoming training 
within his agency. 


Two of the three reported that they would have made changes to the process of their 
participation in order to maximize the potential benefits. Specifically, they would have 
changed the way they selected and prepared staff to participate in the training. 

Because the NYS site was at a different level from the other two sites (e.g., regional 
rather than within a single city), one administrator felt there was not a strong enough 
connection with the schools and would have invited participation from that level. Another 
stated that she would have done more to engage trainees upfront and describe the 
reasons for their participation in the YSP/KSA Demo. She felt it was more "done to 
them" versus having them see the benefits and connections to their work. 


Another administrator would have liked more time to select the modules and suggested 
that another meeting of all the site administrators would have helped them be more 
thoughtful about that selection. She felt their logic model was on target and reflected 
the areas they needed to address, but more thoughtful discussion may have made it 
more reflective of all three agencies' needs and perhaps support "buy-in" for the project 
a bit more. Although there may not have been as much progress on the logic model 
outcomes specifically, all three administrators reported connections across the agencies 
and use of the training materials from the YSP/KSA Demo. 
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Observations Across the Demonstration Sites 


As described previously, each of the demonstration project sites developed a logic 
model including long-term outcomes and the specific practices they wanted to target for 
improvement. The six-month surveys focused more on the practices (behavior change) 
and the 12-month surveys focused on the outcomes (results). Each site’s surveys were 
tailored to these individualized outcomes and practices, however a set of common items 
was included on each of the six-month surveys. These items addressed the frequency 
with which participants across the sites had: 1) used the strategies and/or resources 
themselves, 2) shared the knowledge they gained with others, and 3) connected with 
staff from partner agencies. 


Six-Month Results 


As depicted in Figures 13 and 14, at six-month post-training delivery, the majority of 
participants across the sites were regularly using the strategies and/or resources from 
the trainings in their professional practice (70.21% and 51.06%). The fact that just over 
70% of the participants reported they used the strategies regularly indicates that the 


Figure 13. Cross-site responses to Figure 14. Cross-site responsese to 
"| use strategies | learned during the | use resources provided during the 
sraininias in iny; protessional prawice” trainings in my professional practice. 

(n=47) (neat) 
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4.26% 


a Doing Reguiatly = Doing Regularly 


= Doing Infrequently 


& Doing Infrequent 
or Rarely g q y 


or Rarely 


= Does Not Apply to 


= Does Not Apply to 
My Job ped 


My Job 


trainings were achieving some results across the demonstration sites related to 
changes in staff behavior — individual practice — in providing services to youth. This 
reported use of strategies and resources also emerged as a common theme across all 
of the sites in the administrator interviews. Each noted the benefit of having a set of 
resources readily available to staff as they need them in their daily work. Specific ways 
these resources have been used noted in the interviews were: 1) sharing with others in 
and across the agencies; and 2) using them to develop webinars and other training 
Opportunities. 
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Another common item included in each of the online surveys addressed the frequency 
with which training participants shared the knowledge they gained during the trainings 
with other staff. Figure 15 displays the results from those responses, which indicate 
that more than half of the respondents regularly share the knowledge they gained 


(64%). 


Respondents were also asked to address the frequency with which they communicated 
with staff from the partner agencies attending the training. The majority of respondents 
indicated they did this infrequently (53%). [See Figure 16.] The results of these two 
items indicate that there is slightly more regular sharing and/or connecting within 


agencies rather than cross-agency. 


Figure 15. Cross-site responses to "I 
share knowledge gained during the 
trainings with other staff." (n=47) 
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= Doing 
Regularly 


m= Doing 
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Rarely 


= Does Not Apply 
to My Job 


Figure 16. Cross-site responses to "I 
communicate professionally with staff 
from any of the partnering agencies who 
attended the training." (n=47) 


2% = Doing Regularly 


= Doing 
Infrequently or 
Rarely 


= Does Not Apply 
to My Job 


In addition to the common rating items on each of the online surveys, participants 
across the sites were asked about their progress in implementing the action steps they 
noted immediately following the trainings. Across the three demonstration sites, 
seventy percent (70%) of the trainees reported that they had taken some kind of action 
step since the trainings. As reported in the individual summaries, these actions 
included: a commitment to doing more outreach to employer partners, being more 
intentional about gathering information about the needs of youth with disabilities 
(Baltimore); reaffirming they will use the information from youth to inform the 
development of the individual service plans, sharing the resources with their colleagues 
(Los Angeles); and specifically revisiting the questions used to gather information from 
youth, using the materials from the training modules to assess the quality of their 
practices (New York). The information that a majority of participants (70%) indicated 
that they took these action steps together with the data about individual use of 
strategies and/or resources and sharing of these with others in their agency re-enforce 
that at six months following the trainings, there was influence on individual practices at 
each of the demonstration sites. 


The set of items included in each site’s survey regarding organizational support for 
applying the knowledge gained through the modules were very specific to the site’s 
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logic model and targeted areas for improvement. Therefore, aggregation of that 
information is not possible. 


An analysis of the responses to organizational support items from each site indicated 
strong agreement that participants feel encouraged to apply their learning and improve 
their practices. This is reflected in the average agreement responses to these items 
from each site: 87.2% in Baltimore, 80.3% in Los Angeles, and 86.7% in New York. 


Within each site, responses to the items that solicited information about more general 
aspects of encouragement had higher agreement ratings than those that got into the 
specific improvements. For example, in Baltimore, the items about encouragement to 
generally improve practices had higher agreement (92.9%) than those specifically 
regarding improving services to youth with disabilities (78.6%). In Los Angeles, the 
agreement ranged from 100% for the encouragement to improve the quality of youth 
service plans to 41.2% agreement that the organization has developed a shared 
resource map for working across partner agencies. 


The New York responses follow the same overall pattern where encouragement to 
implement strategies to increase stakeholder involvement received higher agreement 
(100%) than the items focused on implementing these strategies specifically to diverse 
populations (73.3%). In all three sites, this may be reflective of where they are in the 
implementation and application of the learning from the module trainings, i.e., — 
generally able to use and apply the knowledge gained, but not yet having accomplished 
the more targeted improvements. 


As described above, there is a general sense that there is organizational support for 
implementing the strategies learned through the trainings at each of the three 
demonstration sites. Some sites have taken specific steps to incorporate updates and 
learning from the trainings in their regular meetings. The importance of having support 
from supervisors to make connections with partner agency staff and of having time to 
practice individual skills were identified across the sites, but these conditions were not 
met routinely at each of the sites. 


As reflected in the summary of action steps taken by participants at each of the 
demonstration sites, there is a more intentional approach to getting meaningful 
information from youth when developing the individual service plans. This emerged 
as a theme across the two sites (Baltimore and Los Angeles) where more staff work 
directly with youth. This intentionality ranged from revising the assessment process to 
more informal approaches such as staff asking different questions to solicit the 
information from youth. In both cases, this has helped make the process less about the 
paperwork and more about the youth. 


The influence on the participating agencies’ culture of professional development is 
evident at each of the demonstration sites. This includes having a stronger sense of 
quality professional development that is more closely related to staff job responsibilities, 
having the benefit of more in-depth training, and conducting cross-agency professional 


45 


development and training. There were also some common reflections about 
considerations for a more intentional selection of staff to attend trainings and the 
training topics. 


It was evident from the data collected approximately six months following the module 
trainings that, regardless of the areas targeted for improvement, each of the sites had 
some change in individual practices. The participants who responded to the survey felt 
they had organizational support for improving those practices. There was also sharing 
of the knowledge learned within agencies, but this may not have happened as 
frequently across the participating agencies at each demonstration site. This may be a 
reflection of the day-to-day workload and sense of priority being given to implementing 
the strategies within before moving to cross agency work. 


12-Month Results 


Across all three of the demonstration sites, respondents indicated that there was 
achievement of many of the outcomes identified in their individual logic models. There 
was also overall agreement that staff at each of the sites was implementing individual 
practices that reflect quality youth service plans. The group interviews with 
administrators supported the online survey responses and provided some overall 
observations about the benefits of participating in the YSP/KSA Demo. 


Based on the online survey results, respondents were regularly implementing individual 
practices identified in their logic model outcomes. Specifically, Baltimore had an 
average of 66% agreement that this was being done regularly across all of the specific 
practices; Los Angeles had 61% agreement, and in New York 55% of survey 
respondents reported that they implemented the individual practices frequently or all of 
the time. 


The same pattern was evident in the results regarding organizational support. An 
average of 69% of Baltimore survey respondents agreed that their organization's work 
was addressing the logic model outcomes, 73% of Los Angeles agreed regarding their 
organizations, and in New York, 68% agreed that this was the case. While the sites 
varied in the extent to which they achieved the outcomes of their logic models, all made 
strides in making progress toward them. 


The Baltimore site may have made more progress than the others because the partner 
agencies are all funded by the Mayor’s Office of Employment Development. This was 
noted by one of the administrators who described the communication regarding the 
YSP/KSA trainings at the management and staff levels. The staff has made strides in 
reviewing and revising the assessment process and have support from leadership to do 
so. While results show that many of the desired outcomes have been achieved and 
staff are regularly implementing quality individual practices, there is room for continued 
improvement regarding services for youth with disabilities. As one site administrator 
noted, participation in the YSP/KSA Demo has strengthened the culture of professional 
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development and the partner agencies will continue to seek out professional 
development that fits with their philosophy regarding professional development. 


While the Los Angeles site data may have a limited perspective regarding regular 
implementation of individual practices due to the number of respondents who did not 
work directly with youth, the data nonetheless demonstrates that many of the desired 
outcomes where achieved by the partner agencies involved in the YSP/KSA Demo. 
With the exception of those who indicated that the individual practices mentioned were 
not relevant to their job, all respondents either agreed or strongly agreed that they were 
regularly helping youth obtain work experiences relevant to their career interests, 
engaging youth in career exploration activities relevant to their career interests, 
providing information to youth on career pathway options, documenting youth’s career 
interests, using assessment information to develop youths’ individual service plans, 
tailoring plans to each youth’s individual needs and goals, and sharing resources with 
others in the organization. 

In addition, Los Angeles administrators described a strengthened culture of professional 
development as an outcome gained from participation in the YSP/KSA Demo that they 
will continue to support across the partner agencies as they move forward. The more 
regular connections between the partner agency staff was an area identified for further 
improvement, but generally there was more sharing of resources and pooling of 
information across the partners. 


The limited data from the New York State site indicates that there had been 
improvement in engaging new and diverse employer partners and in deepening the 
youth voice in service provision. While there may be more remaining to be done to 
strengthen connections across the partner agencies, respondents to the online survey 
indicated that there was a common language and increased cooperation across the 
partners. Administrators’ interviews reinforced that there have been cross-agency 
connections and noted that this may have happened more at the administrator level. 
Although they did not get as far as they may have liked with progress toward their logic 
model outcomes, the administrators affirmed that the outcomes they identified were the 
"right ones,” and that it was likely that their involvement in the demonstration assisted 
them in preparing for implementation of WIOA. 


The identified improvements for each of the YSP/KSA Demo sites varied, but as the 
results of the twelve-month data collection show, each made strides in achieving the 
outcomes they identified in their logic model. All reported some degree of sharing of the 
resources from the trainings pointing to continued use of the resources and tools 
provided. This was something noted in the group interviews — the usefulness of the 
resources and strategies and the fact that they were being used rather than “put ona 
shelf.” For two of the sites, the culture of professional development was enhanced by 
participation in the YSP/KSA Demo. Even if the professional development activities 
themselves are not always cross-agency, they are sharing information about 
Opportunities that fit with the philosophy they have developed regarding quality 
professional development. Based on the analysis of the online surveys and group 
interviews, all the sites have made improvements since participating in the YSP/KSA 
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Demo trainings — some in individual practice and others in organizational culture — 
which have supported improved services for youth at their agencies. 
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Appendix A: Baltimore Data Collection Protocols 


Six-month Follow-Up Trainee Survey: Baltimore 


1. Since completing the training, have you changed positions, agencies, or 
organizations? If yes, please explain. 


2. In your current position, do you work directly with youth and young adults? 
Yes No 


3. Since completing the training, what if anything have you done to: 
a) improve the quality and individualization of youth service plans? 
b) increase and sustain employer engagement? 


c) improve how you serve and engage youth with disabilities? 


4. Please indicate the extent to which you have done the following since the YSP/KSA 


training: 
Allthe Frequently Infrequently Almost Does 
time Never Not 
Apply 
to My 
Job 
a. | use strategies | learned 
during the trainings in my O O 
professional practice. 
. | use resources provided 
during the trainings in my O O 
professional practice. 
. | share knowledge gained 
during the trainings with O O 
other staff. 
. | communicate professionally 
with staff from any of the q q 
partnering agencies who 
attended the training. 
. | use assessment 
information to develop q q 


youth’s individual service 
plans. 
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f. | tailor the individual service 
plans that | develop with 
youth to each youth’s O O O O O 
individual needs and 
strengths. 


g. | identify and engage new 
employer partners. 


h. | maintain regular contact 
with existing employer =] O O iS XO 
partners. 


i. | incorporate any disability 
related needs of a youth into 
her/his individual service 
plan. 


j. | connect youth to disability 
services available in the O O O O O 
community when needed. 


k. | use universal design 
strategies and 
accommodations to better 
serve youth with disabilities. 


5. Please indicate the extent to which your organization has done the following since 
the YSP KSA training: 


Strongly Agree Disagree Strongly Does 


Agree Disagree Not 
Apply 
to My 
Job 
a. My organization 
encourages employees to 
make changes or 
improvements to our 0 0 0 0 0 
practices based on the 
trainings. 
b. My organization oO 
encourages implementation 
of new strategies based on 0 0 0 0 
the trainings. 
c. My organization 
encourages employees to 
improve the quality of O O O O O 
youth’s individual service 
plans. 
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d. My organization 
encourages employees to 
implement strategies to oO oO oO oO oO 
increase and sustain 
employer engagement. 


e. My organization 
encourages employees to 
improve how they engage 
youth with disabilities. 


6. Recently, you were sent a list of action steps you indicated you would take as a result 
of the trainings. Using that list, please respond to the following: 


a. Have you completed any of the actions you identified as next steps at the 
conclusion of the training sessions? C1 YesU) No 


b. If you responded “yes” to the above, which action steps have you taken? 


c. If you responded “no” to the above, what are any reasons that you did not take 
these actions? 


7. If you have used strategies and/or resources from the trainings, please describe 
what those were and how you used them. 
8. What, if any, changes have you made in how youth are served since your 


organization participated in the trainings? Please describe those. 


9. Please provide any other observations regarding your individual practice and/or your 
organization since completing the trainings. 


Thank you for taking time to share your thoughts! 
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12-Month Follow-Up Survey: Training Participants—Baltimore 


1. 


Since completing the training, have you changed positions, agencies, or 
organizations? If yes, please explain. 


In your current position, do you work directly with youth and young adults? Yes 
No 


. Please indicate the extent to which you are doing the following since the YSP/KSA 


training: 


Strongly Agree Disagree Strongly Does 


Agree Disagree Not 
Apply 
to My 
Job 
| regularly use assessment 
information to develop Z 7 7 Z Z 


youth’s individual service 
plans. 


| regularly tailor the 

individual service plans that 

| develop with youth to each O O O O O 
youth’s individual needs 

and strengths. 


| regularly identify and 


engage new employer O O O O XO 
partners. 

| regularly maintain contact 

with existing employer O O O O O 
partners. 


| regularly incorporate any 
disability related needs of a 
youth into her/his individual 
service plan. 


| regularly connect youth to 
disability services available 
in the community when 
needed. 

| regularly use universal 
design strategies and 
accommodations to better 
serve youth with disabilities. 


BZ 


4. Please provide any additional comments regarding how your individual practice has 
changed, if at all, since the YSP/KSA training. 


5. Please indicate the extent to which you agree or disagree with the following 
statements about your organization’s work since the YSP/KSA training: 


Strongly Agree Disagree Strongly 


Agree Disagree 
The quality of youth service 
plans has improved. O O O O 
Youth service plans have 
become more tailored to 
each youth's individual 0 0 O O 
needs and goals. 
The number of employer 7 q q q 


partners has increased. 
More of our existing 
employer partners have O O O = 
remained engaged. 

More youth with disabilities 
are connected to disability 
services matching their 
individual needs. 

Youth with disabilities are 
better engaged in our 
programs as a result of O O al is 
universal design strategies 

and accommodations. 


6. Please provide any additional comments regarding changes in your organization’s 
practices since the YSP/KSA training. 


7. If your organization has used strategies and/or resources from the trainings, please 
describe what those were and how they have been used. 
8. What, if any, changes have been made in how youth are served since your 


organization participated in the trainings? Please describe those. 


Thank you for taking time to share your thoughts! 
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12-Month Follow-Up Administrator Interview Questions: Baltimore 


¢ How would you describe the quality of youth service plans? 
e Towhat extent are these individualized? 
e Towhat extent are staff using assessment information to develop the plans? 


¢ How well are youth with disabilities being served by your organization? How are 
staff connecting these youth to services matching their needs? 


¢ To what extent are universal design strategies and accommodations being used? 


¢ Have you increased the number of employer partners? If yes, how did you achieve 
this? 


¢ Have you strengthened the relationships with existing employer partners? If yes, 
how did you do this? 


As you reflect on your overall participation in this project, what final thoughts do you 
have on the impact and lessons learned regarding the cross-system professional 
development? 
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Appendix B: Los Angeles Data Collection Protocols 


Six-Month Follow-Up Trainee Survey: Los Angeles 


i. 


Since completing the training, have you changed positions, agencies, or 
organizations? If yes, please explain. 


In your current position, do you work directly with youth or young adults? 


Yes No 


3. Since completing the training, what if anything have you done to: 


a) improve the quality and individualization of youth service plans? 


b) share resources? 


4. Please indicate the extent to which you have done the following since the YSP/KSA 


training: 
Allthe Frequently Infrequently Almost Does 
time Never Not 
Apply 
to My 
Job 
| use strategies | learned 
during the trainings in my O O O XO O 
professional practice. 
| use resources provided 
during the trainings in my O O O O O 
professional practice. 
| share knowledge gained 
during the trainings with other O O O XO XO 
Staff. 
| communicate professionally 
with staff from any of the 
partnering agencies who 0 0 0 0 eB 
attended the training. 
| participate in cross-system 
professional development O O O O O 
activities. 
| connect with at least one of 
the partner agencies who qo q q q q 


attended the YSP/KSA 
training. 
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Allthe Frequently Infrequently Almost Does 
time Never Not 
Apply 
to My 
Job 

| use assessment information 

to develop youth’s individual 

service plans. O O O O O 

| tailor the individual service 

plans that | develop with 

youth to each youth’s 0 0 0 0 0 

individual needs and goals. 

| identify and document 

youth’s career interests. 0 0 0 0 0 

| engage youth in career 

exploration activities relevant O O O a O 

to their career interests. 

| help youth obtain work 

experiences relevant to their O O O O O 

career interests. 

| provide information to youth 

on career pathway options. 0 0 0 O 0 

| share resources with others 

within our organization. O 0 0 0 0 

| share resources with other 

professionals for the partner O O O O O 

agencies. 

| use a resource map to refer 

youth to services. 0 0 0 0 0 

| use a resource map to q q q q q 


increase partnerships. 


5. Please indicate the extent to which your organization has done the following since 


the YSP KSA training: 
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Strongly Agree Disagree Strongly Does 
Agree Disagree Not 
Apply 
to My 
Job 


My organization encourages 
employees to participate in 
cross-system professional 
development activities. 


O O O XO O 


My organization encourages 
employees to connect with the 
partner agencies who 
attended the YSP/KSA. 


My organization encourages 


employees to improve the 
quality of youth’s individual 
service plans. 


My organization encourages 
employees to improve career 
exploration and work 
experience opportunities for 
youth. 


My organization encourages 
employees to provide 
information to youth on career 
pathway options. 


My organization encourages 
employees to identify and 
share resources. 


My organization has 
developed a shared resource 
map with the partner agencies 
who attended the YSP/KSA 
training. 


. Recently, you were sent a list of action steps you indicated you would take as a result 
of the trainings. Using that list, please respond to the following: 


a. Have you completed any of the actions you identified as next steps at the 
conclusion of the training sessions? Yes No 


b. If you responded “yes” to the above, which action steps have you taken? 
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c. If you responded “no” to the above, what are any reasons that you did not take 
these actions? 


7. If you have used strategies and/or resources from the trainings, please describe 
what those were and how you used them. 


8. What, if any, changes have you made in how youth are served since your 
organization participated in the trainings? Please describe those. 


9. Please provide any other observations regarding your individual practice and/or your 
organization since completing the trainings. 


Thank you for taking time to share your thoughts! 
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12-Month Follow-Up Survey: Training Participants—Los Angeles 


1. Since completing the training, have you changed positions, agencies, or 
organizations? If yes, please explain. 


2. In your current position, do you work directly with youth or young adults? Yes 
No 


3. Please indicate the extent to which you are doing the following since the YSP/KSA 
training: 


Strongly Agree Disagree Strongly Does 
Agree Disagree Not 
Apply 
to My 
Job 


| regularly participate in 
cross-system professional O O O O 
development activities. 
| regularly connect with at 
least one of the partner 


agencies who attended the 0 0 0 0 0 
YSP/KSA training. 

| regularly use assessment 

information to develop q q q q q 


youth’s individual service 
plans. 

| regularly tailor the 
individual service plans that 
| develop with youth to O O O O O 
each youth’s individual 

needs and goals. 

| regularly identify and 

document youth’s career O O O EJ EJ 
interests. 

| regularly engage youth in 
career exploration activities 


relevant to their career 0 O 0 0 0 
interests. 

| regularly help youth obtain 

work experiences relevant O O O O O 


to their career interests. 

| regularly provide 

information to youth on O O O O O 
career pathway options. 
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| regularly share resources 

with others within our O O O O O 
organization. 

| regularly share resources 

with other professionals [© O O O O 
for the partner agencies. 


4. Please provide any additional comments regarding your individual practice since the 
YSP/KSA training. 


5. Please indicate the extent to which you agree or disagree with the following 
statements about your organization’s work since the YSP/KSA training: 


Strongly Agree Disagree Strongly 
Agree Disagree 


Our organization offers cross- 

system professional development O ia O XO 
opportunities. 

Our organization has built 

connections with at least one the 


partner agencies who attended the 0 0 0 0 
YSP/KSA training. 

The quality of youth individual 

service plans has become more q q q q 


tailored to each youth’s individual 
needs and goals. 

More youth have been participating 
in work experiences relevant to their O O O E] 
career interests. 

Our organization connects youth to 
a wider variety of internships, 
summer jobs, trainings, service 
projects, and 
subsidized/unsubsidized 
employment opportunities. 

Staff from the three partner 
agencies are sharing resources for QO O O O 
youth more frequently. 

Our organization has works with a 
greater variety of partners. 


6. Please provide any additional comments regarding your organization’s practices 
since the YSP/KSA training. 
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. If your organization has used strategies and/or resources from the trainings, please 
describe what those were and how they have been used. 


. What, if any, changes have been made in how youth are served since your 
organization participated in the trainings? Please describe those. 


Thank you for taking time to share your thoughts! 
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12-Month Follow-Up Administrator Interview Questions: Los Angeles 


How would you describe the quality of youth service plans? 
o Towhat extent are these individualized? 
o Towhat extent are staff using assessment information to develop the 
plans? 


To what extent staff from the partner agencies participating in cross-system 
professional development? Please describe. 


To what extent are staff from the partner agencies connecting, collaborating, and 
cooperating? 


To what extent are youth participating in work experiences relevant to their 
career interests? Success stories/examples? 


Has a resource map been developed? If so, how are staff using it? 


To what extent are staff from the partner agencies sharing resources for youth? 


As you reflect on your overall participation in this project, what final thoughts do you 
have on the impact and lessons learned regarding the cross-system professional 
development? 
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Six-Month Follow-Up Trainee Survey: New York 


1. Since completing the training, have you changed positions, agencies, or 
organizations? If yes, please explain. 


2: 


3. Since completing the training, what if anything have you done to: 


Appendix C: New York Data Collection Protocols 


In your current position, do you work directly with youth? 


a) connect, collaborate, or cooperate with the staff of any of the other agencies? 


b) build cross-agency knowledge around workforce issues and best practice? 
c) increase and strengthen employer engagement? 


d) increase the engagement of stakeholders and partners? 


training: 


. Please indicate the extent to which you have done the following since the YSP KSA 


| use strategies | learned 
during the trainings in my 
professional practice. 


Allthe Frequently Infrequently Almost 


Does 
Not 
Apply 
to My 
Job 


O 


| use resources provided 
during the trainings in my 
professional practice. 


| share knowledge gained 
during the trainings with other 
Staff. 


| communicate professionally 
with staff from any of the 
partnering agencies who 
attended the training. 


| connect with at least one of 
the partner agencies who 
attended the YSP/KSA 
training. 
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Allthe Frequently Infrequently Almost Does 


time Never Not 
Apply 
to My 
Job 
| share resources and 
strategies with staff from at 
least one of the partner O O O O XO 
agencies who attended the 
YSP/KSA training. 
| participate in activities 
designed to build cross- 
agency knowledge around O O O O XO 
workforce issues and best 
practices. 
| identify and engage new 
employer partners or support O O O O O 


others to do so. 

| maintain regular contact with 

existing employer partners or = O O [il = 
support others to do so. 


| identify and engage new 
partners and stakeholders or O O EJ O O 
support others to do so. 


| use new strategies to 
engage existing partners and 
stakeholders or support 
others to do so. 


| in corporate youth voice into 
day-to-day youth activities or O O O O XO 
support others to do so. 


. Please indicate the extent to which your organization has done the following since 
the YSP KSA training: 


Strongly Agree Disagree Strongly Does 
Agree Disagree Not 
Apply 
to My 
Job 


My organization encourages 
employees to connect, [=] O O O O 
collaborate, or cooperate with 
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Strongly Agree Disagree Strongly Does 


Agree Disagree Not 


Apply 
to My 


Job 


the partner agencies who 
attended the YSP/KSA. 


My organization encourages oO 


employees to participate in 
activities designed to build 
cross-agency knowledge 
around workforce issues and 
best practices. 

My organization encourages 
employees to implement 
strategies to increase and 
strengthen employer 
engagement with diverse 
youth populations. 


My organization encourages 
employees to implement 


strategies to increase the oO Ey oO O O 


engagement of stakeholders 
and partners. 


. Recently, you were sent a list of action steps you indicated you would take as a result 


of the trainings. Using that list, please respond to the following: 


a. Have you completed any of the actions you identified as next steps at the 
conclusion of the training sessions? Yes No 


b. If you responded “yes” to the above, which action steps have you taken? 
c. If you responded “no” to the above, what are any reasons that you did not take 
these actions? 


If you have used strategies and/or resources from the trainings, please describe 
what those were and how you used them. 


. What, if any, changes have you made in how youth are served since your 
organization participated in the trainings? Please describe those. 
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9. Please provide any other observations regarding your individual practice and/or your 
organization since completing the trainings. 


Thank you for taking time to share your thoughts! 
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12-Month Follow-up Survey - Training Participants 
New York 


1. Since completing the training, have you changed positions, agencies, or 
organizations? If yes, please explain. 


2. In your Current position, do you work directly with youth or young adults? 
Yes No 
3. Please indicate the extent to which you are doing the following since the 
YSP/KSA training: 
Allthe Frequently Infrequently Almost Does 
time Never Not 
Apply 
to My 
Job 
a. | regularly connect with 
at least one of the 
partner agencies who O O O O O 
attended the YSP/KSA 
training. 


b. | regularly share 
resources and 
strategies with staff 
from atleast one ofthe O O O a O 
partner agencies who 
attended the YSP/KSA 
training. 


c. | regularly participate in 
activities designed to 
build cross-agency 
knowledge around 
workforce issues and 
best practices. 


d. | regularly identify and 
engage new employer 
partners or support 
others to do so. 


e. | regularly maintain 
contact with existing 
employer partners or 
support others to do so. 
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Allthe Frequently Infrequently Almost Does 
time Never Not 
Apply 
to My 
Job 
f. | regularly identify and 
engage new partners 
and stakeholders or 0 0 O 0 0 
support others to do so. 
g. | regularly use new 
strategies to engage 
existing partners and O O O O O 
stakeholders or support 
others to do so. 
h. | regularly incorporate 
youth voice into day-to- q q q 7 q 


day youth activities or 
support others to do so. 


4. Please provide any additional comments regarding your individual practice since the 


YSP/KSA training. 


5. Please indicate the extent to which you agree or disagree with the following 
statements about your organization’s work since the YSP/KSA training: 


Strongly Agree 
Agree 


Disagree Strongly 
Disagree 


a. Professionals from the partner 
agencies who attended the 
YSP/KSA training are regularly 
connecting as a community of 
practice. 


O O 


O O 


b. There is increased cooperation 
between our organization and at 


least one of the partner agencies O 


who attended the YSP/KSA 
training. 


c. Staff from the partner agencies 
have developed a 
shared/common knowledge 
about workforce issues and best 
practices. 


O O 


68 


Strongly Agree Disagree Strongly 


Agree Disagree 
d. The number of employer 
partners has increased. O O O O 
e. Our employer partners are more 
engaged with diverse O O O O 
populations. 
f. Our organization has engaged q q q q 


new partners and stakeholders. 
g. Our organization’s existing 
partners and stakeholders are O O O O 
more deeply engaged. 
h. Our organization has increased 
and deepened youth [a O O O 
engagement and voice. 


6. Please provide any additional comments regarding changes in your 
organization’s practices since the YSP/KSA training. 


7. If your organization has used strategies and/or resources from the trainings, 
please describe what those were and how they were used. 


8. What, if any, changes have been made in how youth are served since your 
organization participated in the trainings? Please describe those. 


Thank you for taking time to share your thoughts! 
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12-Month Follow-Up Administrator Interview Questions: New York 


¢ Have you increased the number of employer partners? If yes, how did you achieve 
this? 


¢ Have you strengthened the relationships with existing employer partners? If yes, 
how did you do this? 


¢ To what extent are staff from the partner agencies connecting, collaborating, and 
cooperating? 


¢ To what extent are youth participating in work experiences relevant to their career 
interests? Success stories/examples? 


¢ To what extent do staff from the partner agencies have shared/common knowledge 
about workforce issues and best practices? 


¢ What, if any, activities have helped over the past year to build shared/common 
knowledge? 


¢ How has your organization increased engagement of partners and stakeholders? 


¢ In what ways are youth being more engaged and having increased voice? Success 
stories/examples? 


As you reflect on your overall participation in this project, what final thoughts do you 
have on the impact and lessons learned regarding the cross-system professional 
development? 
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Appendix D: Results Tables from Six-Month Follow-Up Survey by Site 


Six-Month Follow-Up Survey Item Result Tables (Yes/No & Rated Items Only): Baltimore 


Since completing the training, have you changed positions, agencies, or organizations? 


# Answer Response | % 

1 Yes Zz 13% 

2 \No 13 87% 
Total 15 100% 


In your current position, do you work directly with youth and young adults? 


# _Answer Response % 

ik Yes 11 73% 

2 No 4 27% 
Total ALS) 


Please indicate the extent to which you have done the following since the YSP/KSA trainings: 


# 


Question 


| use strategies | learned 
during the trainings in my 
professional practice. 


| use resources | provided 
during the trainings in my 
professional practice. 


| share knowledge gained 
during the trainings with 
other staff. 


| communicate 
professionally with staff 
from any of the partnering 
agencies who attended 
the training. 


| use assessment 
information to develop 
youth's individual service 
plans. 


All of the 
Time 


Frequently 


13 


12 


Infrequently 


Rarely Does Not Total 


Apply to Responses 
My Job 

0 0 14 

1 0 14 

1 0 14 

2 6) 14 

0 eS) 14 
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# 


10 


41 


Question 


| tailor the individual 
service plans that | 
develop with youth to 
each youth's individual 
needs and strengths. 


| identify and engage new 
employer partners. 


| maintain regular contact 
with existing employer 
partners. 

| incorporate any disability 
related needs of a youth 
into her/his individual 
service plan. 


| connect youth to disability 
services available in the 
community when needed. 


All of the 
Time 


Frequently 


Infrequently 


Rarely Does Not Total 
Apply to Responses 


My Job 
0 3 14 
© Z 14 
- 2 14 
1 =) 14 
2 3 14 
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# Question All of the 
Time 
12 | use universal design 4 


strategies and 
accommodations to better 
serve youth with 
disabilities. 


Frequently 


Infrequently Rarely Does Not 
Apply to 
My Job 


0 Z 4 


Total 
Responses 


14 
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Question 


My organization encourages 
employees to make changes or 
improvements to our practices 
based on the trainings. 


My organization encourages 
implementation of new strategies 
based on the trainings. 


My organization encourages me to 
improve the quality of youth's 
individual service plans. 


My organization encourages me 
to implement strategies to 
increase and sustain employer 
engagement. 


My organization encourages 
me to improve how | engage 
youth with disabilities. 


Strongly Agree 
Agree 


7 6 
8 5 
4 7 
9 4 
4 7 


Disagree 


Strongly 
Disagree 


Please indicate the extent to which your organization has done the following since the YSP/KSA trainings: 


Does Not 


Apply to My 
Job 


0 


Total 


Responses 


14 


14 


14 


14 


14 
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Have you completed any of the actions you identified as next steps at the conclusion of the training sessions? 


Answer Response % 
1 eS 13 93% 
2 No 1 T% 
Total 14 
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Six-Month Follow-Up Survey Item Result Tables (Yes/No & Rated Items Only): Los Angeles 


Since completing the training, have you changed positions, agencies, or organizations? 


# Answer Response 
Yes S) 

2  |No 15 
Total 18 


In your current position, do you work directly with youth and young adults? 


# Answer 
1 Yes 
2 No 

Total 


Please indicate the extent to which you have done the following since the YSP/KSA trainings: 


# Question All of 
the Time 
1 |use strategies | 1 


learned during the 
trainings in my 
professional practice. 
2 |use resources | 0 
provided during the 
trainings in my 
professional practice. 
3 | share knowledge 2 
gained during the 


Response 


Frequently 


Aba 


Infrequently 


Does Not Total 
Apply to My Responses 
Job 


1 18 
1 18 
0 18 
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11 


Question 


trainings with other staff. 


| communicate 
professionally with staff 
from any of the 
partnering agencies 
who attended the 
training. 

| connect with at least 
one of the partner 
agencies who attended 
the YSP/KSA training. 
| participate in cross- 
system professional 
development activities. 
| use assessment 
information to develop 
youth's individual 
service plans. 

| tailor the individual 
service plans that | 
develop with youth to 
each youth's individual 
needs and goals. 

| identify and document 


youth's career interests. 


| engage youth in career 
exploration activities 
relevant to their 
interests. 


All of 
the Time 


Frequently 


Infrequently 


Rarely 


Does Not 


Apply to My Responses 


Job 


Total 


18 


18 


18 


18 


18 


18 


18 
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# Question All of Frequently Infrequently Rarely Does Not Total 


the Time Apply to My Responses 
Job 
12 | help youth obtain work 2 8 O 1 7 18 
experiences relevant to 
their career interests. 
13 | provide information to 3 8 1 1 5 18 
youth on career pathway 
options. 
14 | share resources with 6 8 2 2 O 18 


others within our 
organization. 
15 | share resources with 2 8 S 2 i 18 
other professionals and 
partner agencies. 


16 | use aresource map to 3 5 3 2 3 17 
refer youth to services. 
17 | use aresource map to Z 5 4 4 3 18 


increase partnerships. 


Please indicate the extent to which your organization has done the following since the YSP/KSA trainings: 


Does Not 
# Question Strongly agree Disagree Strongly Total 
Agree Disagree ApplytoMy Responses 
Job 
4 My organization encourages 5 7 3 1 Al 18 


employees participate in 
cross-system professional 
development. 
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Question 


My organization encourages 
employees to connect with the 
partner agencies who attended the 
YSP/KSA. 


My organization encourages 
employees to improve the 
quality of youth's individual 
service plans. 


My organization encourages 
employees to improve career 
exploration and work experience 
Opportunities for youth. 


My organization encourages 
employees to provide information 
to youth on career pathway 
options. 


My organization encourages 
employees to identify and share 
resources. 


Strongly 
Agree 


11 


10 


Agree 


11 


Strongly Does Not Total 
Disagree ApplytoMy Responses 
Job 

1 0 17 

0 0 17 

1 0 18 

1 0 18 

1 0 18 
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# Question 


g My organization has developed 
a shared resource map with the 
partner agencies who attended 
the YSP/KSA training. 


Strongly 
Agree 


Agree 


Disagree 


Strongly 
Disagree 


Does Not 
Apply to My 
Job 


Z 


Total 
Responses 


Lz 
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Have you completed any of the actions you identified as next steps at the conclusion of the training sessions? 


# Answer Response % 
Yes g 50% 

Z No 9 50% 
Total 18 
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Six-Month Follow-Up Survey Item Result Tables (Yes/No & Rated Items Only): New York 


Since completing the training, have you changed positions, agencies, or organizations? 


# Answer Response % 


Yes 7 44% 
2 |No 9 56% 
Total 16 100% 


In your current position, do you work directly with youth and young adults? 


# Answer Response % 


1 |Yes 3 20% 
2 |No 12 80% 
Total 15 100% 


Please indicate the extent to which you have done the following since the YSP/KSA trainings: 


# Question Allofthe Frequently infrequently Rarely 


Time 
1 |use strategies | learned during 1 8 5 a 
the trainings in my professional 
practice. 


Does 
Not 
Apply 
to My 
Job 


0 


Total 
Responses 


15 
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Question 


| use resources | provided during 
the trainings in my professional 
practice. 

| share knowledge gained during 
the trainings with other staff. 

| communicate professionally 
with staff from any of the 
partnering agencies who 
attended the training. 


| connect with at least one of the 
partner agencies who attended 
the YSP/KSA training. 


| share resources and strategies 
with staff form at least one of 
the partner agencies who 
attended the YSP/KSA training. 
| participate in activities 
designed to build cross- 
agency knowledge around 
workforce issues and best 
practices. 

| identify and engage new 
employer partners or support 
others to do so. 

| maintain regular contact with 
existing employer partners or 
support others to do so. 


All of the Frequently 
Time 


1 3 
0 6 
0 4 
al i 
1 3 
0 uk 
0 7 
0 6 


Infrequently Rarely 


9 1 
z 1 
9 1 
3 3 
Fé 2 
6 1 
2 3 
2 Z 


Does 
Not 
Apply 
to My 
Job 


1 


Total 
Responses 


15 


Ie 


15 


AS) 


AS 


1S 


13 


10S) 


84 


# Question All of the | Frequently 
Time 

10 | identify and engage new 0 7 
partners and stakeholders or 
support others to do so. 

11 | use new strategies to engage 0 8 
existing partners and 
stakeholders or support others to 
do so. 

12 | incorporate youth voice into day- 2 S| 


to-day youth activities or support 
others to do so. 


Infrequently Rarely 


Please indicate the extent to which your organization has done the following since the YSP/KSA trainings: 


# Question 


1 My organization encourages 
employees to connect, collaborate, 
or cooperate with the partner 
agencies who attended the 
YSP/KSA training. 


Strongly 
Agree 


Agree 


10 


Disagree 


Strongly Does 


Disagree Not 
Apply 
to My 

Job 
0 1 


Total 
Responses 


15 


Ie 


15 


Total 
Responses 


15 
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# Question 


2 My organization encourages 
employees to participate in 
activities designed to build cross- 
agency knowledge around 
workforce issues and best 
practices. 


3 My organization encourages me to 
implement strategies to increase and 
strengthen employer engagement 
with diverse youth populations. 


4 My organization encourages me 
to implement strategies to 
increase the engagement of 
stakeholders and partners. 


Strongly 
Agree 


Agree Disagree 


9 0 
S 0 
a 0 


Strongly 
Disagree 


Total 
Responses 


15 


15 


15 
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Have you completed any of the actions you identified as next steps at the conclusion of the training sessions? 


# | Answer Response % 

1) res 11 73% 

2 No 4 27% 
Total 15 100% 
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